Delete Catalog Versions Automatically

You can use the config service to delete catalog versions automatically, by enabling the automaticVersionDeletion and setting the numberOfVersionsToKeep at the time of your catalog creation or during an update at a later stage.

If set, when the number of versions in a catalog exceeds the value set for numberOfVersionsToKeep, a new minimum version will be set for the catalog and all prior versions will be deleted. Any catalog versions as recent as or more recent than your minimum version will not be deleted. Similarly, any partition metadata and data that is still used in current, non-deleted versions will not be deleted so that the non-deleted versions remain functional.

Example: Given a versioned layer with 10 versions, you can configure the catalog by setting numberOfVersionsToKeep=10, to store a maximum of 10 versions. On the next increment to version 11, a job will asynchronously trigger the deletion of version 1. This process will repeat for every new commit.

Enable automatic version deletion

This procedure to enable automatic deletion of catalog versions is done, by setting the numberOfVersionsToKeep using the config service. For complete information on using the config service, see the Config API Reference.

1. Obtain an authorization token. For instructions, see the Identity & Access Management Guide.
2. Use the API Lookup service to get the API endpoint for the config v1 API to update the catalog. For instructions, see the API Lookup Developer's Guide.
3. Set the numberOfVersionsToKeep for the catalog's configuration using the request below:

   PUT /catalogs/<catalogHrn> HTTP/1.1
   Host: <Hostname for the config API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   {
     ...
     NOTE: remainder of the catalog configuration hidden for clarity
     ...

     "automaticVersionDeletion": {
       "numberOfVersionsToKeep": 10
     }
   }

1. The request returns 202 Accepted.

Enable automatic version deletion on catalog creation

Similarly, the automaticVersionDeletion can be set on the catalog creation operation. For more information on creating a catalog, see the Config API Reference.

Disable automatic version deletion

To stop the automated deletion of catalog versions, use the config API.

1. Obtain an authorization token. For instructions, see the Identity & Access Management Guide.
2. Use the API Lookup service to get the API endpoint for the config v1 API to update the catalog. For instructions, see the API Lookup Developer's Guide.

3. Disable the automatic version deletion using this request:

   DELETE ​/catalogs​/{catalogHrn}​/automaticVersionDeletion HTTP/1.1
   Host: <Hostname for the config API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   

4. The request returns 202 Accepted.