Publish data to a volatile layer

To publish data to a volatile layer, use the REST APIs publish (for metadata) and volatile-blob (for data).

You can optionally store metadata for every partition that contains data. Uploading metadata is recommended for querying purposes. For example, you can use the metadata API to query for partitions that contain data, or for partitions that contain data with specific metadata properties. If you include metadata in a volatile layer, it should be published with every new partition that contains data. This is usually done when publishing data to the layer for the first time. You should also include metadata when you publish data for new partitions. Similarly, you should remove metadata when you delete partition data. Every partition's metadata contains a data handle which is used to access or modify the data itself data using the volatile-blob API. If multiple partitions contain the same data, they can use the same data handle. After metadata is published for partitions, the respective data handles can be used to update the data repeatedly without requiring subsequent metadata publications.

The typical flow for publishing metadata for partitions consists of the following steps.

1. Obtain an authorization token.
2. Get API base URL.
3. Initialize a publication.
4. Generate metadata.
5. Upload metadata.
6. Submit the publication.
7. Wait until the publication completes.

The typical flow for publishing data to partitions consists of following steps.

1. Get API base URL.
2. Upload/Delete data.

Publishing metadata

Obtain an authorization token

Obtain an authorization token for your HTTP requests. For instructions, see the Identity & Access Management Guide.

Get API base URL

Use the API Lookup service to get the base URL for the publish v2 API for the catalog you want to publish to. For instructions, see the API Lookup Developer's Guide.

Initialize a publication

Initialize a publication by requesting a publication ID. The publication ID is a unique identifier for the set of metadata publications you are going to make to a volatile layer.

To initialize your publication, use the publish API:

POST /<Base path for the publish API from the API Lookup Service>/publications HTTP/1.1
Host: <Hostname for the publish API from the API Lookup Service>
Authorization: Bearer <Authorization Token>
Content-Type: application/json
Cache-Control: no-cache

{
 "layerIds": [
   "<Layer to Publish To>"
 ]
}

The response will contain a publication ID. For example:

{
    "id": "abc1234567-xy90-11aa-abc123-abc123456789",
    ...
}

Generate metadata

The way you generate metadata depends on whether you are adding or deleting partitions.

Generate metadata for new partitions

At a minimum, partition metadata for new partitions must consist of:

• A partition ID
• The data handle that will be used to upload and consume the data

You are responsible for generating the data handle for each of the new partitions. The data handle can be anything you choose, but there are some requirements. The data handle:

• Can be any unique number or a hash of the content or metadata. However, partitions can use the same data handle if each of the partitions contain the same data.
• Must be unique within the volatile layer.

For a complete description of the metadata fields you can set, see the API Reference information about the publish API.

Generate metadata for deleted partitions

Partition metadata for deleted partitions must consist of:

• Empty data handles for the deleted partitions. Publishing empty data handles deletes the metadata for the deleted partitions.
• The partition ID of each deleted partition.

Upload metadata

After all your metadata has been generated and aggregated, it's time to upload your metadata. Aggregated metadata contains all partitions being added and removed. Partitions being removed will have empty data handle. To upload metadata, use the publish API:

POST /<Base path for the publish API from the API Lookup Service>/layers/<Layer ID>/publications/<Publication Id>/partitions HTTP/1.1
Host: <Hostname for the publish API from the API Lookup Service>
Authorization: Bearer <Authorization Token>
Content-Type: application/json
Cache-Control: no-cache
{ "partitions":
  [
    {
      "dataHandle": "<Data Handle 1>",
      "partition": "<Partition 1>",
      "checksum": "<Checksum>"
    },
    {
      "dataHandle": "<Data Handle 2>",
      "partition": "<Partition 2>",
      "checksum": "<Checksum>"
    },
    {
      "dataHandle": "<Data Handle 3>",
      "partition": "<Partition 3>",
      "checksum": "<Checksum>"
    }
  ]
}

Example for deleting the metadata of a partition:

To delete metadata, use the publish API and follow the same steps for publishing metadata, but simply submit empty dataHandles for the partitions you would like to delete. For example:

  POST /<Base path for the publish API from the API Lookup Service>/layers/<Layer ID>/publications/<Publication Id>/partitions HTTP/1.1
  Host: <Hostname for the publish API from the API Lookup Service>
  Authorization: Bearer <Authorization Token>
  Content-Type: application/json
  Cache-Control: no-cache
  {
    "partitions": [
      {
        "partition": "partition01"
        "dataHandle": ""
      }
     ]
  }

You can delete and add partitions in a single publication.

Submit the publication

After all your metadata has been uploaded, you should submit the publication to indicate that the metadata is uploaded. Submitting the publication is a recommended best practice. In addition, submitting a publication is required to support the deprecated use of metadata API to get volatile metadata by version.

To submit your publication, use the publish API.

PUT /<Base path for the publish API from the API Lookup Service>/publications/abc1234567-xy90-11aa-abc123-abc123456789 HTTP/1.1
Host: <Hostname for the publish API from the API Lookup Service>
Authorization: Bearer <Authorization Token>
Cache-Control: no-cache

After your publication has been submitted, the HERE platform processes the publication. Depending on the current volume of publications, and the size of your publication, processing can take from a few minutes, to a few hours. You can check the status of your publication using the publish API.

  GET /<Base path for the publish API from the API Lookup Service>/publications/<Publication Id> HTTP/1.1
  Host: <Hostname for the publish API from the API Lookup Service>
  Authorization: Bearer <Authorization Token>
  Cache-Control: no-cache

For publish v2, the response of a completed publication returns HTTP status 200 OK and contains the value succeeded in the state field in the response body. For example:

  {
    "layerDetails": [],
    "catalogVersion": 394,
    "id": "v2-1ca3aee1-1c3f-4a98-8288-ecfed7466133",
    "catalogId": "my-catalog",
    "details": {
        "expires": 1526769797871,
        "state": "succeeded",
        "modified": 1526510600520,
        "message": "Publication has succeeded",
        "started": 1526510597871
    },
    "layerIds": [
        "versioned-layer"
    ],
    "type": "partition",
    "versionDependencies": [
        {
            "hrn": "hrn:my-company:data:::my-catalog",
            "version": 395,
            "direct": true
        }
    ]
  }

Waiting for completion is an important part of the versioned publication process because you cannot start a new publication until the previously-finalized publication has completed. Any attempt to start a new publication will result in an HTTP 409 Conflict response.

Publishing data

Get API base URL

Use the API Lookup service to get the base URL for the volatile-blob API for the catalog you want to publish to. For instructions, see the API Lookup Developer's Guide.

Upload/Delete data

Now that publishing metadata is complete, you can start uploading data using the volatile-blob API. Put the data you want to upload in the request body. Each time data is uploaded for a given data handle, the existing data is overwritten with the updated data.

You can upload data many times using the same metadata publication. However, a new metadata publication is required if:

• The set of data partitions has changed (likely due to change in coverage).
• You are including partition checksums. You need to use the publish API to specify the updated partition checksums.

You need to know the data handle of the partition you want to update. You can upload and re-upload data for the initialized partitions (partitions which have metadata) using their existing data handles without updating their metadata.

PUT /<Base path for the volatile-blob API from the API Lookup Service>/layers/<Layer ID>/data/<Data Handle> HTTP/1.1
Host: <Hostname for the volatile-blob API from the API Lookup Service>
Authorization: Bearer <Authorization Token>
Content-Type: application/octet-stream
Cache-Control: no-cache

<Data to Upload>

If you are deleting data in a partition, you need the data handle. Ensure you do not loose the data handle if you are deleting metadata before the data itself.