Publish to a Stream Layer

There are multiple ways to send data to a stream layer depending on how much data you want to upload in a request.

Publish up to 20 MB of Data in a Request

If you are sending data into a stream layer and each payload is less than 20 MB, use the ingest API. There are two endpoints you can use to publish a stream of data, one that is data-agnostic, and one that is designed for SDII Message List.

  1. Obtain an authorization token to use with your HTTP requests. For instructions, see the Authentication and Authorization API Guide.
  2. Use the API Lookup service to get the base URL for the ingest API for the catalog you want to publish to. For instructions, see the API Lookup Developer's Guide.
  3. Send the data using either the data-agnostic endpoint or the SDII Message List endpoint if you are streaming a list of SDII messages.

Note

If the ingested data is less than 1 MB in size, the data is ingested directly into the stream layer. If the ingested data is equal to or greater than 1 MB, the data is first uploaded to blob storage which will result in a charge for blob storage. The data will be available via the blob API for the duration of the stream time-to-live (TTL) configured for the stream layer. After the TTL expires, the data is deleted from Blob storage.

Data-Agnostic

  POST /<Base path for the ingest API from the API Lookup Service>/layers/<Layer ID> HTTP/1.1
  Host: <Hostname for the ingest API from the API Lookup Service>
  Authorization: Bearer <Authorization Token>
  Content-Type: <Content type configured for the target layer>
  Cache-Control: no-cache

  <Byte Array of Data>

SDII Message List

  POST /<Base path for the ingest API from the API Lookup Service>/layers/<Layer ID>/sdiiMessageList HTTP/1.1
  Host: <Hostname for the ingest API from the API Lookup Service>
  Authorization: Bearer <Authorization Token>
  Content-Type: application/x-protobuf
  Cache-Control: no-cache

  <Byte Array of Data in SDII MessageList Protobuf Format>

Note

The SDII Message List endpoint splits the input message list into individual SDII messages, so when consumers read the data, it must be read as individual SDII messages.

The data-agnostic request returns a trace ID. You can use this ID to track your request and identify the message in the catalog. The SDII Message List request returns a list of IDs for the individual messages. You can optionally specify a trace ID in the request. For more information, see the API Reference.

Publish More Than 20 MB of Data in a Request

Publishing data that is more than 20 MB is slightly more complicated than uploading smaller amounts of data. To publish more than 20 MB of data you must upload the data using the blob API, then publish the data using the publish API. You can use this this process for data uploads of any size, but since it involves using two APIs instead of one, you may want to only follow this process if your data is more than 20 MB.

  1. Obtain an authorization token for your HTTP requests.

    For instructions on obtaining an authorization token, see:

    Authentication and Authorization API Guide

  2. Get the API base URLs for the publish and blob APIs for the catalog you want to publish to.

    To get the API base URLs, use the API Lookup service. For instructions, see:

    API Lookup Developer's Guide

  3. Initialize the publication.

    To initialize the publication, specify the layers you want to publish to. When a publication is initialized, you receive a publication token which you then use to publish your metadata using the publish API along with information about your newly initialized publication.

    To initialize your publication use the publish API:

    POST /<Base path for the publish API from the API Lookup Service>/publications HTTP/1.1
    Host: <Hostname for the publish API from the API Lookup Service>
    Authorization: Bearer <Authorization Token>
    Content-Type: application/json
    Cache-Control: no-cache
    
    {
      "layerIds": [
        "<Layer to Publish To>"
      ]
    }
    

    The response will contain a publication ID. For example:

    {
        "id": "abc1234567-xy90-11aa-abc123-abc123456789",
        ...
    }
    

    Note

    The publish API only supports one active publication at a time for versioned layers. If you initialize a publication while another publisher is actively publishing, the active publication is canceled. If the existing publication has been submitted but not yet completed, you get a 409 Conflict error when you try to initialize.

  4. Generate a data handle.

    In order to upload data using the blob API, you need to generate a unique identifier for the data. This is referred to as the data handle.

    When generating a data handle, be aware of the following:

    • The data handle can be anything you choose, but it must be unique within the layer, across all versions.
    • The data handle can be any unique number or a hash of the content or metadata.
    • Once a data handle is used to upload data, the data handle cannot be reused.
    • If a data upload fails because the data handle is already in use, your retry logic must generate a new data handle for the upload.

    You should create logic to manage data handle conflicts. For example, if you restart your processing and attempt to re-upload data to an existing data handle, you will receive an error and your upload will fail.

    You can check to see if a data handle already exists by using the blob API.

    HEAD /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/data/<Data Handle> HTTP/1.1
    Host: <Hostname for the blob API from the API Lookup Service>
    Authorization: Bearer <Authorization Token>
    Cache-Control: no-cache
    

    A successful response indicates that the specified data handle exists.

    Now you can use the blob API to upload data using the data handle you generated by initializing a multipart upload, uploading the data parts, and completing the multipart upload.

  5. Initialize a multipart upload for the data handle.

    To initialize a multipart upload for the data handle, use this request:

    POST /<Base path for the blob API from the API Lookup Service>/layers/<Layer ID>/data/<Data Handle>/multiparts HTTP/1.1
    Host: <Hostname for the blob API from the API Lookup Service>
    Authorization: Bearer <Authorization Token>
    Content-Type: application/json
    Cache-Control: no-cache
    
    {
      "contentType": "application/octet-stream"
    }
    

    The response contains the URLs to manage the upload.

     {
       "links": {
         "uploadPart": {
           "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=/parts",
           "method": "POST"
         },
         "complete": {
           "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=",
           "method": "PUT"
         },
         "status": {
           "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=",
           "method": "GET"
         },
         "delete": {
           "href": "https://blobstore.data.api.platform.here.com/blobstore/v1/catalogs/documentation-test/layers/versioned-layer-1/data/DataHandleA/multiparts/eyJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGljYXRpb24vanNvbiIsInVwbG9hZElkIjoiZS42eXhqRGdkSEttMlVvcThFYUZGUUk0V2tyV01naFNrcVJiaGc5ZFp4eEVWY1pFUVBZdDBMeXlLQ1NRS1BUcHlwWGg4eEVoTlBFbnJLYnptTnRmbWVUdnJpY1plZi45M2VybWU3S2NwZHJiT2lyQjdTS1Bod2FUSTg5ZmpRNHoifQ=",
           "method": "DELETE"
         }
       }
     }
    
  6. Upload data parts.

    Now that you have initialized the multipart upload, you can upload your data using the POST method's URL from the previous step, which is named uploadPart.

    If you are uploading more than 50 MB of data, the data must be split into parts and you must upload each part individually. The minimum size of each part is 5 MB, except the last part which does not have a minimum size. The maximum amount of data in a part is 5 GB. You must also provide a part number in the query parameter partNumber, which will be used later to reassemble your data.

    If you are uploading less than 50 MB of data, you can upload the data in a single part and there is no minimum size.

    POST <Path for uploadPart from prior step> HTTP/1.1
    Host: <Hostname for uploadPart from prior step>
    Authorization: Bearer <Authorization Token>
    Content-Type: application/octet-stream
    Cache-Control: no-cache
    
    <Data to Upload>
    

    The response contains an etag header which is the ID of your uploaded part. Collect these and couple them with their associated part number, as they will be necessary to complete your multipart upload.

    Note

    We recommend that your application includes retry logic for handling HTTP 5xx errors. Use exponential backoff in the retry logic.

  7. Complete the multipart upload.

    To complete the multipart upload, provide the etags and part numbers of all the uploaded parts using the URL named complete to construct this request:

    PUT <Path for complete from prior step> HTTP/1.1
    Host: <Hostname for complete from prior step>
    Authorization: Bearer <Authorization Token>
    Content-Type: application/json
    Cache-Control: no-cache
    
    {
      "parts": [
        {
          "etag": "yJvYmplY3RJZCI6ImVyaWtzSGFuZGxlQSIsImNvbnRlbnRUeXBlIjoiYXBwbGlj"
          "number": 1
        },
        {
          "etag": "1h2h3k4j5h6ZCI6ImVyaWtzSGFuZGxlQadjiwo234ADF2323k34H5heoH543Hr"
          "number": 2
        }
      ]
    }
    
  8. Generate metadata.

    When all data parts have been uploaded, you need to generate the metadata which will be used to represent your data inside the HERE platform. At a minimum, your metadata must consist of a partition ID and the data handle you used to upload your data. For a complete description of the fields you can set, see the API reference for the publish API:

    API Reference

    Once you have generated the metadata, aggregate it somewhere so that once all your data has been uploaded, and all your metadata has been generated, you can proceed to the next step and upload your metadata.

  9. Upload metadata.

    To upload metadata, use the publish API. For example:

    POST /<Base path for the publish API from the API Lookup Service>/layers/<Layer ID>/publications/<Publication Id>/partitions HTTP/1.1
    Host: <Hostname for the publish API from the API Lookup Service>
    Authorization: Bearer <Authorization Token>
    Content-Type: application/json
    Cache-Control: no-cache
    {
      "partitions": [
        {
          "compressedDataSize": 0,
          "dataHandle": "DataHandleA",
          "dataSize": 0,
          "partition": "partition01"
        }
       ]
    }
    

    Note

    The maximum number of partitions you can upload and publish in one request is 1,000. If you have more than 1,000 partitions you want to upload, upload the data for the first 1,000 partitions then upload the metadata for those 1,000 partitions. Then, continue to the next set of partitions. Do not wait until all data is uploaded before uploading metadata. Doing so will result in poor performance. Instead, upload data, then metadata, and repeat as needed until all your data and corresponding metadata is uploaded.

  10. Delete metadata

    To delete metadata, use the publish API and follow the same steps for publishing metadata, but simply submit empty dataHandles for the partitions you would like to delete. For example:

    POST /<Base path for the publish API from the API Lookup Service>/layers/<Layer ID>/publications/<Publication Id>/partitions HTTP/1.1
    Host: <Hostname for the publish API from the API Lookup Service>
    Authorization: Bearer <Authorization Token>
    Content-Type: application/json
    Cache-Control: no-cache
    {
    "partitions": [
      {
        "partition": "partition01"
        "dataHandle": ""
      }
     ]
    }
    

    You can delete and add partitions in a single publication.

  11. Submit the publication.

    To submit your publication, use the publish API.

    PUT /<Base path for the publish API from the API Lookup Service>/publications/<Publication ID> HTTP/1.1
    Host: <Hostname for the publish API from the API Lookup Service>
    Authorization: Bearer <Authorization Token>
    Cache-Control: no-cache
    

    After submitting the publication, the metadata processing will begin. You will be unable to modify or interrupt the publication process so make sure that you are ready before submitting the publication.

  12. Wait for the publication to complete.

    After your publication has been submitted, the HERE platform processes the publication. Depending on the current volume of publications, and the size of your publication, processing can take from a few minutes, to a few hours. You can check the status of your publication using the publish API.

    GET /<Base path for the publish API from the API Lookup Service>/publications/<Publication Id> HTTP/1.1
    Host: <Hostname for the publish API from the API Lookup Service>
    Authorization: Bearer <Authorization Token>
    Cache-Control: no-cache
    

    For publish v2, the response of a completed publication returns HTTP status 200 OK and contains the value succeeded in the state field in the response body. For example:

    {
    "layerDetails": [],
    "catalogVersion": 394,
    "id": "v2-1ca3aee1-1c3f-4a98-8288-ecfed7466133",
    "catalogId": "my-catalog",
    "details": {
        "expires": 1526769797871,
        "state": "succeeded",
        "modified": 1526510600520,
        "message": "Publication has succeeded",
        "started": 1526510597871
    },
    "layerIds": [
        "versioned-layer"
    ],
    "type": "partition",
    "versionDependencies": [
        {
            "hrn": "hrn:my-company:data:::my-catalog",
            "version": 395,
            "direct": true
        }
    ]
    }
    

Note

You must check for the value succeeded in the state field rather than only relying on the HTTP status as the HTTP status does not correlate to the publication state. If the status is 'failed', a new publication needs to be initialized and metadata re-uploaded after the issue detailed in the failure message has been addressed. Any uploaded data and data handles can be reused.

Waiting for completion is an important part of the versioned publication process because you cannot start a new publication until the previously-finalized publication has completed. Any attempt to start a new publication will result in an HTTP 409 Conflict response.

results matching ""

    No results matching ""