Publish data to an object store layer

To publish data to an object store layer, use the REST API blob v2.

The typical publication flow consists of the following steps.

1. Obtain an authorization token.
2. Get API base URLs.
3. Choose a Key.
4. Upload data.

These steps are described below.

1. Obtain an authorization token for your HTTP requests.

   For instructions on obtaining an authorization token, see the

   Identity & Access Management Guide.

2. Get the API base URL for the blob v2 APIs for the catalog you want to publish to.

   To get the API base URLs, use the API Lookup service. For instructions, see the

   API Lookup Developer's Guide.

3. Choose a key

   In order to upload data using the blob object store API, you need to choose an identifier, referred to as the object_key.

   • Can be any of the following characters a-zA-Z0-9.[]=(){}/_-`.
   • A '/' (slash) will be interpreted by a separator to define folder-like structure.

4. Upload data

   To upload data for an object key.

   PUT /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/keys/<Object Key> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: <Content-Type>
   Cache-Control: no-cache
   
   <Data to Upload>
   

   Note

   We recommend that your application includes retry logic for handling HTTP 5xx errors. Use exponential backoff in the retry logic.

The single part upload can be used for objects up to 192 MB. Use multipart upload for any data objects larger than 192 MB.

For more information on the Object store size limits, see Workspace limits and quotas.

1. Initialize a multipart upload.
2. Upload data parts.
3. Complete the multipart upload.
4. Wait for the status.

These steps are described below.

1. Initialize a multipart upload.

   To initialize a multipart upload for an object key.

   POST /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/keys/<Object Key> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   
   {
       "contentType": "<Content-Type>",
       "contentEncoding": "<Content-Encoding>"
   }
   

   The response contains the multipartToken used to manage this multipart upload for the provided object key.

   {
      "multipartToken": "7767cf80..."
   }
   

2. Upload data parts.

   Now that you have initialized the multipart upload, you can upload your data using the multipartToken received from the previous step.

   If you are uploading more than 192 MB of data, the data must be split into parts and you must upload each part individually. The minimum size of each part is 5 MB, except the last part which does not have a minimum size. The maximum amount of data in a part is 5 GB. You must also provide a part number in the query parameter partNumber, which will be used later to reassemble your data.

   If you are uploading less than 192 MB of data, you can upload the data in a single part and there is no minimum size.

   POST /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/keysMultipart/<MultipartToken>/parts?partNumber=<Part Number> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: <Content-Type>
   Cache-Control: no-cache
   
   <Data to Upload>
   

   The response contains the ID of the uploaded part. Collect these and couple them with their associated part number, as they will be necessary to complete your multipart upload.

   { "id": "YTZmMDhkN..." }
   

   Note

   We recommend that your application includes retry logic for handling HTTP 5xx errors. Use exponential backoff in the retry logic.

3. Complete the multipart upload.

   To complete the multipart upload, provide the ID and part numbers of all the uploaded parts using the URL named complete to construct this request:

   POST /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/keysMultipart/<MultipartToken> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   
   {
     "parts": [
       {
         "id": "yJvYmplY..."
         "number": 1
       },
       {
         "id": "1h2h3k4..."
         "number": 2
       }
     ]
   }
   

4. Wait for the status.

   After complete a multipart upload, you need to wait for completion by requesting the status of the upload.

   GET /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/keysMultipart/<MultipartToken> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   
   {
       "status": "completed"
   }
   

To publish data to an object store layer, use the REST API blob v2.

The typical publication flow consists of the following steps.

1. Obtain an authorization token.
2. Get API base URLs.
3. Choose a Key.
4. Upload data.

These steps are described below.

1. Obtain an authorization token for your HTTP requests.

   For instructions on obtaining an authorization token, see the

   Identity & Access Management Guide

2. Get the API base URLs for the blob v2 APIs for the catalog you want to publish to.

   To get the API base URLs, use the API Lookup service. For instructions, see the API Lookup Developer's Guide.

1. Choose a key

   In order to upload data using the blob object store API, you need to choose an identifier, referred to as the object_key.

   • Can be any of the following characters a-zA-Z0-9.[]=(){}/_-`.
   • A '/' (slash) will be interpreted by a separator to define folder-like structure.

2. Upload data

   To upload data for a object key.

   PUT /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/keys/<Object Key> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: <Content-Type>
   Cache-Control: no-cache
   
   <Data to Upload>
   

   Note

   We recommend that your application includes retry logic for handling HTTP 5xx errors. Use exponential backoff in the retry logic.

The single part upload can be used for objects up to 192 MB. Use multipart upload for any data objects larger than 192 MB.

1. Initialize a multipart upload.
2. Upload data parts.
3. Complete the multipart upload.
4. Wait for the status.

These steps are described below.

1. Initialize a multipart upload.

   To initialize a multipart upload for an object key.

   POST /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/keys/<Object Key> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   
   {
       "contentType": "<Content-Type>",
       "contentEncoding": "<Content-Encoding>"
   }
   

   The response contains the multipartToken used to manage this multipart upload for the provided object key.

   {
      "multipartToken": "7767cf80..."
   }
   

2. Upload data parts.

   Now that you have initialized the multipart upload, you can upload your data using the multipartToken received from the previous step.

   If you are uploading more than 192 MB of data, the data must be split into parts and you must upload each part individually. The minimum size of each part is 5 MB, except the last part which does not have a minimum size. The maximum amount of data in a part is 5 GB. You must also provide a part number in the query parameter partNumber, which will be used later to reassemble your data.

   If you are uploading less than 192 MB of data, you can upload the data in a single part and there is no minimum size.

   POST /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/keysMultipart/<MultipartToken>/parts?partNumber=<Part Number> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: <Content-Type>
   Cache-Control: no-cache
   
   <Data to Upload>
   

   The response contains the ID of the uploaded part. Collect these and couple them with their associated part number, as they will be necessary to complete your multipart upload.

   { "id": "YTZmMDhkN..." }
   

   Note

   We recommend that your application includes retry logic for handling HTTP 5xx errors. Use exponential backoff in the retry logic.

3. Complete the multipart upload.

   To complete the multipart upload, provide the ID and part numbers of all the uploaded parts using the URL named complete to construct this request:

   POST /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/keysMultipart/<MultipartToken> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   
   {
     "parts": [
       {
         "id": "yJvYmplY..."
         "number": 1
       },
       {
         "id": "1h2h3k4..."
         "number": 2
       }
     ]
   }
   

4. Wait for the status.

   After complete a multipart upload, you need to wait for completion by requesting the status of the upload.

   GET /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/keysMultipart/<MultipartToken> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   
   {
       "status": "completed"
   }