Delete Catalog Versions Automatically
You can use the config
service to delete catalog versions automatically, by enabling the automaticVersionDeletion
and setting the numberOfVersionsToKeep
at the time of your catalog creation or during an update at a later stage.
Warning
Deleting catalog versions will delete partition metadata AND data associated with those versions, impacting all versioned layers in the catalog. The partition metadata and data will be irrecoverable. Any partition metadata and data that is still used in current, non-deleted versions will not be deleted so that the non-deleted versions remain functional.
If set, when the number of versions in a catalog exceeds the value set for numberOfVersionsToKeep
, a new minimum version will be set for the catalog and all prior versions will be deleted. Any catalog versions as recent as or more recent than your minimum version will not be deleted. Similarly, any partition metadata and data that is still used in current, non-deleted versions will not be deleted so that the non-deleted versions remain functional.
Example: Given a versioned layer with 10 versions, you can configure the catalog by setting numberOfVersionsToKeep=10
, to store a maximum of 10 versions. On the next increment to version 11, a job will asynchronously trigger the deletion of version 1. This process will repeat for every new commit.
Note
The actual data deletion process will be executed asynchronously, so that the request is not blocked by the internal processing of data. In other words, the data deletion process is eventually consistent. The physical metadata and data deletion may take up to three days and billing will continue for that period of time.
Enable automatic version deletion
This procedure to enable automatic deletion of catalog versions is done, by setting the numberOfVersionsToKeep
using the config
service. For complete information on using the config
service, see the Config API Reference.
- Obtain an authorization token. For instructions, see the Identity & Access Management Guide.
- Use the API Lookup service to get the API endpoint for the
config
v1
API to update the catalog. For instructions, see the API Lookup Developer's Guide. - Set the
numberOfVersionsToKeep
for the catalog's configuration using the request below:
Note
A maximum value of 1000 is allowed.
PUT /catalogs/<catalogHrn> HTTP/1.1
Host: <Hostname for the config API from the API Lookup Service>
Authorization: Bearer <Authorization Token>
Content-Type: application/json
{
...
NOTE: remainder of the catalog configuration hidden for clarity
...
"automaticVersionDeletion": {
"numberOfVersionsToKeep": 10
}
}
- The request returns
202 Accepted
.
Enable automatic version deletion on catalog creation
Similarly, the automaticVersionDeletion
can be set on the catalog creation operation. For more information on creating a catalog, see the Config API Reference.
Disable automatic version deletion
To stop the automated deletion of catalog versions, use the config
API.
- Obtain an authorization token. For instructions, see the Identity & Access Management Guide.
- Use the API Lookup service to get the API endpoint for the
config
v1
API to update the catalog. For instructions, see the API Lookup Developer's Guide. -
Disable the automatic version deletion using this request:
DELETE /catalogs/{catalogHrn}/automaticVersionDeletion HTTP/1.1
Host: <Hostname for the config API from the API Lookup Service>
Authorization: Bearer <Authorization Token>
Content-Type: application/json
-
The request returns 202 Accepted
.