Publish data to a stream layer

There are multiple ways to send data to a stream layer depending on how much data you want to upload in a request.

• Publish up to 20 MB of data in a request
• Publish more than 20 MB of data in a request

Publish up to 20 MB of data in a request

If you are sending data into a stream layer and each payload is less than 20 MB, use the ingest API. There are multiple endpoints you can use to publish a stream of data:

• data-agnostic endpoint, this endpoint automatically upload content to blob API if payload is bigger than 1 MB
• endpoint that is designed for SDII Message List.
• partitions endpoint, allows publish of stream messages incl. metadata like partitions ID, crc, checksum, etc.

1. Obtain an authorization token to use with your HTTP requests. For instructions, see the Identity & Access Management Guide.
2. Use the API Lookup service to get the base URL for the ingest v1 API for the catalog you want to publish to. For instructions, see the API Lookup Developer's Guide.
3. Send the data using either the data-agnostic endpoint or the SDII Message List endpoint if you are streaming a list of SDII messages.

Data-Agnostic

  POST /<Base path for the ingest API from the API Lookup Service>/layers/<Layer ID> HTTP/1.1
  Host: <Hostname for the ingest API from the API Lookup Service>
  Authorization: Bearer <Authorization Token>
  Content-Type: <Content type configured for the target layer>
  Cache-Control: no-cache

  <Byte Array of Data>

SDII Message List

  POST /<Base path for the ingest API from the API Lookup Service>/layers/<Layer ID>/sdiiMessageList HTTP/1.1
  Host: <Hostname for the ingest API from the API Lookup Service>
  Authorization: Bearer <Authorization Token>
  Content-Type: application/x-protobuf
  Cache-Control: no-cache

  <Byte Array of Data in SDII MessageList Protobuf Format>

Partitions

  POST /<Base path for the ingest API from the API Lookup Service>/layers/<Layer ID>/partitions HTTP/1.1
  Host: <Hostname for the ingest API from the API Lookup Service>
  Authorization: Bearer <Authorization Token>
  Content-Type: application/json
  Cache-Control: no-cache

  {
     "partitions": [
       {
         "data": "eyJhaWQiOiJ0ZXN0LWFwcC1pZC0xIn0",
         "partition": "partition01",
         "checksum": "291f66029c232400e3403cd6e9cfd36e",
         "crc": "c3f276d7"
       }
      ]
  }

Requests to the data-agnostic and partitions endpoint return a trace ID. You can use this ID to track your request and identify the message in the catalog. The SDII Message List request returns a list of IDs for the individual messages. You can optionally specify a trace ID in the request. For more information, see the API Reference.

Publish more than 20 MB of data in a request

Publishing data blobs that are larger than 20 MB is slightly more complicated than uploading smaller amounts of data. To publish more than 20 MB of data, you must upload the data using the blob API, and then publish the data using the ingest publish partitions API. You can use this process for data uploads of any size, but since it involves the use of two APIs instead of one, you may want to follow this process only if your data is larger than 20 MB.

1. Obtain an authorization token for your HTTP requests.

   For instructions on obtaining an authorization token, see:

   Identity & Access Management Guide

2. Get the API base URLs for the ingest and blob APIs for the catalog you want to publish to.

   To get the API base URLs, use the API Lookup service. For instructions, see:

   API Lookup Developer's Guide

3. Generate a data handle.

   In order to upload data using the blob API, you need to generate a unique identifier for the data. This is referred to as the data handle.

   When generating a data handle, be aware of the following:

   • The data handle can be anything you choose, but it must be unique within the layer, across all versions.
   • The data handle can be any unique number or a hash of the content or metadata.
   • Once a data handle is used to upload data, the data handle cannot be reused.
   • If a data upload fails because the data handle is already in use, your retry logic must generate a new data handle for the upload.

   You should create logic to manage data handle conflicts. For example, if you restart your processing and attempt to re-upload data to an existing data handle, you will receive an error and your upload will fail.

   You can check to see if a data handle already exists by using the blob API.

   HEAD /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/data/<Data Handle> HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Cache-Control: no-cache
   

   A successful response indicates that the specified data handle exists.

   Now you can use the blob API to upload data using the data handle you generated by initializing a multipart upload, uploading the data parts, and completing the multipart upload.

4. Initialize a multipart upload for the data handle.

   To initialize a multipart upload for the data handle, use this request:

   POST /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/data/<Data Handle>/multiparts HTTP/1.1
   Host: <Hostname for the blob API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   
   {
     "contentType": "application/octet-stream"
   }
   

   The response contains the URLs to manage the upload.

    {
      "links": {
        "uploadPart": {
          "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=/parts",
          "method": "POST"
        },
        "complete": {
          "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=",
          "method": "PUT"
        },
        "status": {
          "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=",
          "method": "GET"
        },
        "delete": {
          "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=",
          "method": "DELETE"
        }
      }
    }
   

5. Upload data parts.

   Now that you have initialized the multipart upload, you can upload your data using the POST method's URL from the previous step, which is named uploadPart.

   If you are uploading more than 50 MB of data, the data must be split into parts and you must upload each part individually. The minimum size of each part is 5 MB, except the last part which does not have a minimum size. The maximum amount of data in a part is 5 GB. You must also provide a part number in the query parameter partNumber, which will be used later to reassemble your data.

   If you are uploading less than 50 MB of data, you can upload the data in a single part and there is no minimum size.

   POST <Path for uploadPart from prior step> HTTP/1.1
   Host: <Hostname for uploadPart from prior step>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/octet-stream
   Cache-Control: no-cache
   
   <Data to Upload>
   

   The response contains an etag header which is the ID of your uploaded part. Collect these and couple them with their associated part number, as they will be necessary to complete your multipart upload.

   Note

   We recommend that your application includes retry logic for handling HTTP 5xx errors. Use exponential backoff in the retry logic.

6. Complete the multipart upload.

   To complete the multipart upload, provide the etags and part numbers of all the uploaded parts using the URL named complete to construct this request:

   PUT <Path for complete from prior step> HTTP/1.1
   Host: <Hostname for complete from prior step>
   Authorization: Bearer <Authorization Token>
   Content-Type: application/json
   Cache-Control: no-cache
   
   {
     "parts": [
       {
         "etag": "yJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGlj"
         "number": 1
       },
       {
         "etag": "1h2h3k4j5h6ZCI6ImVyaWtzSGFuZGxlQadjiwo234ADF2323k34H5heoH543Hr"
         "number": 2
       }
     ]
   }
   

7. Generate metadata.

   When all data parts have been uploaded, you need to generate the metadata which will be used to represent your data inside the HERE platform. At a minimum, your metadata must consist of a partition ID and the data handle you used to upload your data. For a complete description of the fields you can set, see the API reference for the ingest API:

   API Reference

   Once you have generated the metadata, aggregate it somewhere so that once all your data has been uploaded, and all your metadata has been generated, you can proceed to the next step and upload your metadata.

8. Upload metadata.

   To upload metadata, use the ingest API. For example:

   ``` POST //layers//partitions HTTP/1.1 Host: Authorization: Bearer Content-Type: application/json Cache-Control: no-cache

{ "partitions": [ { "compressedDataSize": 0, "dataHandle": "DataHandleA", "dataSize": 0, "partition": "partition01", "checksum": "291f66029c232400e3403cd6e9cfd36e", "crc": "c3f276d7" }, { "compressedDataSize": 0, "dataHandle": "DataHandleB", "dataSize": 0, "partition": "partition02", "checksum": "291f66029c232400e3403cd6e9cfd36f", "crc": "c3f276d4" }, ... ] } ```