Publish to an index layer

An index layer is an index of the catalog’s data. One use of an index layer is to archive data from a stream layer so you can query it. For more information, see Archive stream data.

To publish data to an index layer, use the REST API index.

1. Obtain an authorization token to use with your HTTP requests. For instructions, see the Identity & Access Management Guide.
2. Use the API Lookup service to get the base URL for the blob v1 and index v1 API for the catalog you want to write to. For instructions on how to use the API Lookup service, see the API Lookup Developer's Guide.
3. Generate a UUID for your data and upload it via the blob API. The UUID must consist of only lowercase characters and be a valid UUID. For more information on the blob API, see the API Reference.

4. Send the index data to the layer using the index API using the same id that was generated in step 3. For example:

   POST /<Base path for the index API from the API Lookup Service>/layers/<Layer ID> HTTP/1.1
   Host: <Hostname for the index API from the API Lookup Service>
   Authorization: Bearer <Authorization Token>
   Content-Type: <Content type configured for the target layer>
   Cache-Control: no-cache
   
   [
    {
      "checksum": "28271214-1532-4cb3-9cd7-35bef1735055",
      "fields": {
        "ingestionTime": 1552383031000,
        "tile": 23618359
      },
      "id": "e9e05a2b-25d1-415d-bc6a-14a1be626c9a",
      "metadata": {},
      "size": 155
    }
   ]
   

   Attribute value validation rules

The values of the attributes in the request body above must conform to the following rules:

1. Id attribute must comply with the UUID format defined as: [0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}
2. Maximum size of the index record is 256B. The size of the index records is calculated as a sum of its parts using the following rules:
   • All user defined fields are taken into account when calculating the record size
   • All meta attributes are taken into account when calculating the record size
   • The sizes of the fields/attributes depend on their types:
     • bool counts as 1 byte
     • integer (deprecated) & long count as 8 bytes.
     • timestamp, heretile & timewindow count as 8 bytes
     • strings sizes are equal to the number of bytes
3. Timewindow attribute value is a long number in milliseconds from the Epoch. It is required and cannot be null. The Index service will truncate the value based on the "duration" of the timewindow attribute before storage.

Index service POST request constraints

The following constraints are enforced by the index service:

1. The POST request to the index service can't contain the undefined indexing attribute (represented inside "fields"). However the extraneous attributes at the top level of the POST request are being ignored. For example, the extra "greeting" field is added, but index layer will filter it out and it will persist the record without "greeting".

    {
      "checksum": "4d1e358e-b547-478c-abd0-3abee9ac71e8",
      "fields": {
        "ingestionTime": 123,
        "tileId": null
      },
      "id": "75420ce9-e4d0-4d74-9c7a-19810f99a3a9",
      "metadata": {
        "ingestionTime": "1532018660873"
      },
      "size": 100,
      "greeting": "hello"
    }
   

2. For user defined attributes (except timewindow field), if it is not provided, it will be considered as null, and will be stored as null.

3. Maximum number of the index records to be inserted in one Index POST (Insert) API call is 2000.

   For more information on the index API, see the API Reference.