Publish data to a versioned layer

To publish data to one or more versioned layers, the data must all be published together in a single publication so that it can be versioned together. Using the REST APIs publish v2 (for metadata) and blob v1 (for data) to upload and publish data to versioned layers.

The typical publication flow consists of the following steps.

1. Obtain an authorization token.
2. Get API base URLs.
3. Initialize a publication.
4. Generate a data handle.
5. Initialize a multipart upload.
6. Upload data parts.
7. Complete the multipart upload.
8. Wait for the status.
9. Generate metadata.
10. Upload metadata.
11. Delete metadata.
12. Submit the publication.
13. Wait for the publication to complete.

These steps are described below.

1. Obtain an authorization token for your HTTP requests.

   For instructions on obtaining an authorization token, see:

   Identity & Access Management Guide

2. Get the API base URLs for the publish and blob APIs for the catalog you want to publish to.

   To get the API base URLs, use the API Lookup service. For instructions, see:

   API Lookup Developer's Guide

3. Initialize the publication.

   To initialize the publication, specify the layers you want to publish to. When a publication is initialized, you receive a publication token which you then use to publish your metadata using the publish API along with information about your newly initialized publication.

   To initialize your publication use the publish API:

   POST /<Base path for the publish API from the API Lookup Service>/publications HTTP/1.1
   Host: <Hostname for the publish API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   
   {
     "layerIds": [
       "<Layer to Publish To>"
     ]
   }
   

   The response will contain a publication ID. For example:

   {
       "id": "abc1234567-xy90-11aa-abc123-abc123456789",
       ...
   }
   

   Note

   The publish API only supports one active publication at a time for versioned layers. If you initialize a publication while another publisher is actively publishing, the active publication is canceled. If the existing publication has been submitted but not yet completed, you get a 409 Conflict error when you try to initialize.

4. Generate a data handle.

   In order to upload data using the blob API, you need to generate a unique identifier for the data. This is referred to as the data handle.

   When generating a data handle, be aware of the following:

   • The data handle can be anything you choose, but it must be unique within the layer, across all versions.
   • The data handle can be any unique number or a hash of the content or metadata.
   • Once a data handle is used to upload data, the data handle cannot be reused.
   • If a data upload fails because the data handle is already in use, your retry logic must generate a new data handle for the upload.

   You should create logic to manage data handle conflicts. For example, if you restart your processing and attempt to re-upload data to an existing data handle, you will receive an error and your upload will fail.

   You can check to see if a data handle already exists by using the blob API.

   HEAD /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/data/<Data Handle> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Cache-Control: no-cache
   

   A successful response indicates that the specified data handle exists.

   Now you can use the blob API to upload data using the data handle you generated by initializing a multipart upload, uploading the data parts, and completing the multipart upload.

5. Initialize a multipart upload for the data handle.

   To initialize a multipart upload for the data handle, use this request:

   POST /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/data/<Data Handle>/multiparts HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   
   {
     "contentType": "application/octet-stream"
   }
   

   The response contains the URLs to manage the upload.

    {
      "links": {
        "uploadPart": {
          "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=/parts",
          "method": "POST"
        },
        "complete": {
          "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=",
          "method": "PUT"
        },
        "status": {
          "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=",
          "method": "GET"
        },
        "delete": {
          "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=",
          "method": "DELETE"
        }
      }
    }
   

6. Upload data parts.

   Now that you have initialized the multipart upload, you can upload your data using the POST method's URL from the previous step, which is named uploadPart.

   If you are uploading more than 50 MB of data, the data must be split into parts and you must upload each part individually. The minimum size of each part is 5 MB, except the last part which does not have a minimum size. The maximum amount of data in a part is 5 GB. You must also provide a part number in the query parameter partNumber, which will be used later to reassemble your data.

   If you are uploading less than 50 MB of data, you can upload the data in a single part and there is no minimum size.

   POST <Path for uploadPart from prior step> HTTP/1.1
   Host: <Hostname for uploadPart from prior step>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/octet-stream
   Cache-Control: no-cache
   
   <Data to Upload>
   

   The response contains an etag header which is the ID of your uploaded part. Collect these and couple them with their associated part number, as they will be necessary to complete your multipart upload.

   Note

   We recommend that your application includes retry logic for handling HTTP 5xx errors. Use exponential backoff in the retry logic.

7. Complete the multipart upload.

   To complete the multipart upload, provide the etags and part numbers of all the uploaded parts using the URL named complete to construct this request:

   PUT <Path for complete from prior step> HTTP/1.1
   Host: <Hostname for complete from prior step>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   
   {
     "parts": [
       {
         "etag": "yJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGlj"
         "number": 1
       },
       {
         "etag": "1h2h3k4j5h6ZCI6ImVyaWtzSGFuZGxlQadjiwo234ADF2323k34H5heoH543Hr"
         "number": 2
       }
     ]
   }
   

8. Generate metadata.

   When all data parts have been uploaded, you need to generate the metadata which will be used to represent your data inside the HERE platform. At a minimum, your metadata must consist of a partition ID and the data handle you used to upload your data. For a complete description of the fields you can set, see the API reference for the publish API:

   API Reference

   Once you have generated the metadata, aggregate it somewhere so that once all your data has been uploaded, and all your metadata has been generated, you can proceed to the next step and upload your metadata.

9. Upload metadata.

   To upload metadata, use the publish API. For example:

   POST /<Base path for the publish API from the API Lookup Service>/layers/<Layer ID>/publications/<Publication Id>/partitions HTTP/1.1
   Host: <Hostname for the publish API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   {
     "partitions": [
       {
         "compressedDataSize": 0,
         "dataHandle": "DataHandleA",
         "dataSize": 0,
         "partition": "partition01"
       }
      ]
   }
   

   Note

   The maximum number of partitions you can upload and publish in one request is 1,000. If you have more than 1,000 partitions you want to upload, upload the data for the first 1,000 partitions then upload the metadata for those 1,000 partitions. Then, continue to the next set of partitions. Do not wait until all data is uploaded before uploading metadata. Doing so will result in poor performance. Instead, upload data, then metadata, and repeat as needed until all your data and corresponding metadata is uploaded.

10. Delete metadata

    To delete metadata, use the publish API and follow the same steps for publishing metadata, but simply submit empty dataHandles for the partitions you would like to delete. For example:

    POST /<Base path for the publish API from the API Lookup Service>/layers/<Layer ID>/publications/<Publication Id>/partitions HTTP/1.1
    Host: <Hostname for the publish API from the API Lookup Service>
    Authorization: Bearer <Authorization Token>
    Content-Type: application/json
    Cache-Control: no-cache
    {
    "partitions": [
      {
        "partition": "partition01"
        "dataHandle": ""
      }
     ]
    }
    

    You can delete and add partitions in a single publication.

11. Submit the publication.

    To submit your publication, use the publish API.

    PUT /<Base path for the publish API from the API Lookup Service>/publications/<Publication ID> HTTP/1.1
    Host: <Hostname for the publish API from the API Lookup Service>
    Authorization: Bearer <Authorization Token>
    Cache-Control: no-cache
    

    After submitting the publication, the metadata processing will begin. You will be unable to modify or interrupt the publication process so make sure that you are ready before submitting the publication.

12. Wait for the publication to complete.

    After your publication has been submitted, the HERE platform processes the publication. Depending on the current volume of publications, and the size of your publication, processing can take from a few minutes, to a few hours. You can check the status of your publication using the publish API.

    GET /<Base path for the publish API from the API Lookup Service>/publications/<Publication Id> HTTP/1.1
    Host: <Hostname for the publish API from the API Lookup Service>
    Authorization: Bearer <Authorization Token>
    Cache-Control: no-cache
    

    For publish v2, the response of a completed publication returns HTTP status 200 OK and contains the value succeeded in the state field in the response body. For example:

    {
    "layerDetails": [],
    "catalogVersion": 394,
    "id": "v2-1ca3aee1-1c3f-4a98-8288-ecfed7466133",
    "catalogId": "my-catalog",
    "details": {
        "expires": 1526769797871,
        "state": "succeeded",
        "modified": 1526510600520,
        "message": "Publication has succeeded",
        "started": 1526510597871
    },
    "layerIds": [
        "versioned-layer"
    ],
    "type": "partition",
    "versionDependencies": [
        {
            "hrn": "hrn:my-company:data:::my-catalog",
            "version": 395,
            "direct": true
        }
    ]
    }
    

Waiting for completion is an important part of the versioned publication process because you cannot start a new publication until the previously-finalized publication has completed. Any attempt to start a new publication will result in an HTTP 409 Conflict response.