Delete catalog versions to manage storage costs and enable lifecycle management of versioned layers. Catalog version deletion does not impact any other layer types stored in a catalog, only versioned layers. Catalog version deletion safely removes data from versioned layers only and in a way that doesn't break dependencies which exist between different versioned layers in a single catalog. Before deleting versions, consider reviewing the usage information for the versions to make sure no processes, such as pipelines, are depending on the deleted versions. If you delete a catalog version that is being used by a process like a pipeline, you will break the process. For information on viewing the usage information, please see the Logs, Monitoring and Alerts User Guide.
To delete versions, use the
PublishApi provided by the
data-client module. The only required parameter is the
HRN of the catalog:
val publishApi = DataClient().publishApi(catalogHrn) publishApi.setMinimumVersion(minimumVersion = 2)
// create publishApi for a catalog PublishApi publishApi = DataClient.get(myActorSystem).publishApi(catalogHrn); publishApi.setMinimumVersion(5L);
Deleting Catalog versions will delete partition metadata AND data associated with those versions, impacting all versioned layers in the catalog. The partition metadata and data will be irrecoverable. Any partition metadata and data from deleted versions that is still used in current, non-deleted versions will not be deleted so that non-deleted versions remain functional.