Create a catalog

If you have data you want to bring to the HERE platform, you need to create a catalog to contain the data. A catalog is a collection of data that is logically managed as a single set. Catalogs can contain layers that change together and depend on each other. For example, a catalog may contain a map that includes layers for road attributes, topology, and signs.

This topic provides an overview of how to create a catalog using the REST service config v1. For complete information on using the config service, see the API Reference.

(Optional) Regional data storage selection

You can select the region your data is stored in when creating single region and multi-region catalogs using either the config API request or the Choose where you want your data stored field on the Add new catalog interface when creating a catalog.

Selecting closer regions helps in mitigating cross-region latencies when writing and reading data in that region. You can also select the backup region when creating a multi-region catalog.

The following limitations apply when selecting a region for data storage:

• Selecting the region your data is stored in is currently limited to US West and EU West regions. The US West region currently supports the following layer types:
  • Stream Layers
  • Versioned layers
  • Volatile layers
• Regional data storage selection is only available to newly created catalogs and cannot be applied to existing catalogs.
• The region you select during catalog creation cannot be changed after the catalog is created. To change this configuration, the catalog must be deleted and then recreated.

(Optional) Multi-region replication for additional data durability

You can select the multi-region feature using either the config API request or the Replicate your data in a backup region checkbox on the Add new catalog interface when creating a catalog. When you create a multi-region catalog, the platform automatically performs all of the necessary tasks to propagate ongoing data changes to a backup region.

In the event that your region goes down, your data will be available via the backup region, but you should expect higher latencies depending upon where your data requests originate.

If you enable multi-region replication, you’ll be charged for additional storage capacity and data I/O:

• Your storage charges (Blob, Volatile, Stream, Stream TTL, Metadata) will double to reflect the storage of data in a backup region.
• Your transfer charges (Data IO) will increase 2.5 to 4 times, depending upon the size of the objects you’re uploading: less for fewer large objects, more for many small objects.

Multi-region replication is currently supported in the following layer types and operations:

• Reads and writes against versioned layers.
• Reads and writes against volatile layers.
• Reads and writes against stream layers.

Since only versioned, volatile, and stream layer operations are supported currently, multi-region catalogs can only have versioned, volatile, and stream layers added to them.

Due to the different use cases supported by the different layer types, different consistency guarantees and data replication strategies apply:

• For versioned layers:

  • HERE Workspace uses a synchronous data replication process. This replication process adheres to read-after-write consistency where data is acknowledged only when it is successfully written to both your selected and backup regions. The synchronous replication process means that writes to versioned layers are possible only when both your selected region and backup regions are available. During a regional failover event, it will not be possible to write any new data while your region is down. All data already written prior to the incident will be available to read in either region.
  • There is additional latency when the platform copies data into the backup region, which is also impacted by the size of the data.

• For volatile layers:

  • HERE Workspace supports eventual consistency guarantees with asynchronous copying of data to another region. Data is acknowledged when written to your region, and then it is written to the backup region as long as connectivity exists. If the platform cannot write to the backup region when your region is down, data loss will occur.
  • For more information about volatile layer data redundancy, see Volatile layer settings.
  • The replication process should not adversely impact write latency. but there can be a delay (replication time) for data to appear in the backup region, depending on internet connectivity and the size of the data.

• For stream layers:
  • Data is replicated asynchronously to the backup region once the data is written to and acknowledged in the region. Data written to a region, but not yet replicated to the backup region before a regional incident/failover event, will be delivered once the region recovers. Any new data written after a failover event will be written to the backup region – as the new region.
  • The replication process should not adversely impact write latency, but there will be replication time which depends on throughput, connectivity to the other region, and the size of the data.

To enable multi-region replication, you must specify a replication region when you create the catalog. This multi-region configuration is immutable, so once a catalog is created as a multi-region catalog, you cannot change it to be a single region catalog.

To specify a replication region, use the replication block in the payload of the config API request when creating the catalog. For complete information on using the config service, see the API Reference.

Once you've created a multi-region catalog, you can get the API base URLs for each region's catalog by specifying the region query parameter to the API Lookup service.

Create a catalog

Once you have decided on the kind of layers you want the catalog to contain, you are ready to create a catalog.

To create a catalog, follow these steps:

1. Obtain an OAuth token. For instructions, see the Identity & Access Management Guide.
2. Use the API Lookup service to get the API endpoint for the config v1 API. For instructions, see the API Lookup Developer's Guide.
3. Use this request to the config API to create the catalog. This request creates a catalog containing one new layer, but you can create more than one layer if needed. Typically you create new layers in the same request that creates the catalog, although it is possible to create a catalog with no layers, and create layers separately.

  POST /<Base path for the config API from the API Lookup Service>/catalogs HTTP/1.1
  Host: <Hostname for the config API from the API Lookup Service>
  Content-Type: application/json
  Authorization: Bearer <Authorization Token>
  Cache-Control: no-cache
  {
    "description": "<Describe the content and purpose of the catalog>",
    "id": "<A unique identifier>",
    "name": "<Catalog name>",
    "notifications": {
    "enabled": false
    },
    "summary": "<A short description of the content and purpose of the catalog>",
    "layers": [
      {
        "id": "<A unique ID for the layer>",
        "name": "<A user-friendly name to use for display purposes>",
        "summary": "<A short description of the content and purpose of the layer>",
        "description": "<A long description of the content and purpose of the layer>",
        "schema": {
            "hrn": "<The schema's HERE Resource Name>"
        },
        "contentType": "<The media type>",
        "contentEncoding": "<Whether to compress data>",
        "partitioning": {
          "scheme": "<The method to use to divide data in the layer into partitions>",
        },
        "layerType": "<The type of layer: versioned, volatile, or stream>",
        "volume": {
          "volumeType": "durable",
          "maxMemoryPolicy": "failOnWrite"
        }
      }
    ]
  }

You can specify configuration settings for volatile and stream layers. Versioned layers have no configuration settings. For more information, see:

• Configure stream layer settings
• Configure versioned layer settings
• Configure volatile layer settings
• Configure index layer settings
• Configure object store layer settings

• Configure interactive map layer settings

  The response element href contains a URL where you can get the status of the catalog creation process. For example:

  {
  "title": "Catalog Creation",
  "href": "https://config.data.api.platform.hereolp.cn/config/v1/status/da167243-13cd-1312-1b72-2z5d2dea7460",
  "type": "urn:olp-types:status"
  }
  

  The URL returns the status while the catalog is being created. For example:

  {
  "status": "pending"
  }
  

  If you try creating a catalog using an ID that already exists, you will get a 409 Conflict response.

Once the catalog is created, the URL returns the catalog and layer metadata:

{
    "id": "<The identifier you specified in the create request>",
    "hrn": "<The HERE Resource Name of the catalog>",
    "name": "<The catalog name you specified in the request>",
    "summary": "<The summary you specified in the request>",
    "description": "<The description you specified in the request>",
    "owner": {
        "creator": {
            "id": "<The App ID associated with the credentials used in the create request> "
        },
        "organisation": {
            "id": "<The realm of the user ID used in the create request>"
        }
    },
    "tags": [],
    "billingTags": [],
    "created": "<The time stamp indicating when the catalog was created>",
    "layers": [
        {
            "id": "<The layer ID>",
            "hrn": "<The HERE Resource Name of the layer>",
            "name": "<The layer name>",
            "summary": "<The summary description of the purpose and content of the layer>",
            "owner": {
                "creator": {
                    "id": "<The App ID used to create the layer>"
                },
                "organisation": {
                    "id": "<The realm in which the layer was created>"
                }
            },
            "partitioningScheme": "<The partitioning scheme of the layer, HEREtile or generic>",
            "partitioning": {
                "tileLevels": [
                    0
                ],
                "scheme": "heretile"
            },
            "contentType": "application/json",
            "contentEncoding": "gzip",
            "volume": {
                "volumeType": "durable"
            },
            "tags": [],
            "billingTags": [],
            "created": "2018-01-29T16:16:55.402Z",
            "layerType": "<The layer type: versioned, volatile, stream, or objectstore>"
        }
    ],
    "version": 0,
    "notifications": {
        "enabled": false
    }
}

With this metadata, specifically the layer HRNs, you can publish data to the catalog.

Catalog creation example

Here is an example of how to create a catalog. In this example, a new catalog named "My Traffic Catalog" with one layer named "Example Layer". They layer is a versioned layer.

POST /config/v1/catalogs HTTP/1.1
Host: config.data.api.platform.hereolp.cn
Content-Type: application/json
Authorization: Bearer <Authorization Token>
Cache-Control: no-cache

{
  "description": "This is a test catalog.",
  "id": "my-catalog",
  "name": "My Traffic Catalog",
  "notifications": {
  "enabled": false
  },
  "summary": "This is a test catalog.",
  "layers": [
    {
      "id": "my-layer",
      "name": "Example Layer",
      "summary": "The traffic incident layer contains information about events that are affecting the flow of traffic or that may be important for drivers to know.",
      "description": "This layer provides aggregated information about traffic incidents, including the type and location of each traffic incident, status, start and end time, and other relevant data. This data is useful to dynamically optimize route calculations. It also provides access to real-time traffic flow data in including information on speed and congestion for specific regions.",
      "schema": {
          "hrn": "hrn:example:schema:::traffic-schema"
      },
      "contentType": "application/json",
      "contentEncoding": "",
      "partitioning": {
        "scheme": "heretile"
      },
      "layerType": "versioned",
      "volume": {
        "volumeType": "durable"
      }
    }
  ]
}

curl -X POST https://config.data.api.platform.hereolp.cn/config/v1/catalogs \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <Authorization Token>' \
-H 'Cache-Control: no-cache' \
-d '{
  "description": "This is a test catalog.",
  "id": "my-catalog",
  "name": "My Traffic Catalog",
  "notifications": {
  "enabled": false
  },
  "summary": "This is a test catalog.",
  "layers": [
    {
      "id": "my-layer",
      "name": "Example Layer",
      "summary": "The traffic incident layer contains information about events that are affecting the flow of traffic or that may be important for drivers to know.",
      "description": "This layer provides aggregated information about traffic incidents, including the type and location of each traffic incident, status, start and end time, and other relevant data. This data is useful to dynamically optimize route calculations. It also provides access to real-time traffic flow data in including information on speed and congestion for specific regions.",
      "schema": {
          "hrn": "hrn:example:schema:::traffic-schema"
      },
      "contentType": "application/json",
      "contentEncoding": "",
      "partitioning": {
        "scheme": "heretile"
      },
      "layerType": "versioned",
      "volume": {
        "volumeType": "durable"
      }
    }
  ]
}'

Once the catalog is created, a request to the status URL returns:

{
    "id": "my-catalog",
    "hrn": "<Catalog HRN>",
    "name": "My Traffic Catalog",
    "summary": "This is a test catalog.",
    "description": "This is a test catalog.",
    "owner": {
        "creator": {
            "id": "Mejk6DMxAq7kKI5lt2T2"
        },
        "organisation": {
            "id": "olp-here-test"
        }
    },
    "tags": [],
    "billingTags": [],
    "created": "2018-01-29T16:16:55.402Z",
    "layers": [
        {
            "id": "my-layer",
            "hrn": "<Catalog HRN>",
            "name": "Example Layer",
            "summary": "The traffic incident layer contains information about events that are affecting the flow of traffic or that may be important for drivers to know.",
            "description": "This layer provides aggregated information about traffic incidents, including the type and location of each traffic incident, status, start and end time, and other relevant data. This data is useful to dynamically optimize route calculations. It also provides access to real-time traffic flow data in including information on speed and congestion for specific regions.",
            "owner": {
                "creator": {
                    "id": "Mejk09M4Aq71KI5l1t2T2"
                },
                "organisation": {
                    "id": "olp-here-test"
                }
            },
            "partitioningScheme": "heretile",
            "partitioning": {
                "tileLevels": [
                    0
                ],
                "scheme": "heretile"
            },
            "contentType": "application/json",
            "contentEncoding": "gzip",
            "volume": {
                "volumeType": "durable"
            },
            "tags": [],
            "billingTags": [],
            "created": "2018-01-29T16:16:55.402Z",
            "layerType": "versioned"
        }
    ],
    "version": 0,
    "notifications": {
        "enabled": false
    }
}

Choose a layer type

The following table provides a summary of the characteristics of each layer type.

For more information about layer types, see Layers.