Update layer configurations

You can use the Data API to reconfigure data layers through a series of REST requests. The process of reconfiguring a layer varies depending on the type of layer you want to update.

Updates to layers and catalogs are asynchronuous and will not cause any downtime. For more information, see Create a catalog

1. Obtain an authorization token. For more information, see the Identity & Access Management Guide.
2. Use the API Lookup service to get the API endpoint for the config v1 API. For more information, see the API Lookup Developer's Guide.

3. Use this request to the config API to update the catalog and layer. This request updates the catalog with the new fields.

   Note

   For complete information on using the config service, see the API Reference.

   PUT /<Base path for the config API from the API Lookup Service>/catalogs/<Catalog HRN> HTTP/1.1
   Host: <Hostname for the config API from the API Lookup Service>
   Content-Type: application/json
   Authorization: Bearer <Authorization Token>
   Cache-Control: no-cache
   {
    "description": "<Describe the content and purpose of the catalog>",
    "id": "<A unique identifier>",
    "name": "<Catalog name>",
    "notifications": {
    "enabled": false
    },
    "summary": "<A short description of the content and purpose of the catalog>",
    "layers": [
      {
        "id": "<A unique ID for the layer>",
        "name": "<A user-friendly name to use for display purposes>",
        "summary": "<A short description of the content and purpose of the layer>",
        "description": "<A long description of the content and purpose of the layer>",
        "schema": {
            "hrn": "<The schema's HERE Resource Name>"
        },
        "contentType": "<The media type>",
        "contentEncoding": "<Whether to compress data>",
        "partitioning": {
          "scheme": "<The method to use to divide data in the layer into partitions>",
        },
        "layerType": "<The type of layer: index, versioned, volatile, or stream>",
        "volume": {
          "volumeType": <durable, volatile>,
          "maxMemoryPolicy": "failOnWrite"
        }
      }
    ]
   }
   

Partial update to a layer configuration

You can use the Data API to reconfigure data layers through REST requests using the PATCH method. See also: PATCH Method for HTTP

1. Obtain an authorization token. For more information, see the Identity & Access Management Guide.
2. Use the API Lookup service to get the API endpoint for the config v1 API. For more information, see the API Lookup Developer's Guide.

3. Use this request to the config API to update layer only. This request updates the layer for mutable fields.

   Note

   For more information on using the config service, see the API Reference.

   PATCH /<Base path for the config API from the API Lookup Service>/catalogs/<Catalog HRN>/layers/<Layer ID> HTTP/1.1
   Host: <Hostname for the config API from the API Lookup Service>
   Content-Type: application/json
   Authorization: Bearer <Authorization Token>
   Cache-Control: no-cache
   {
    "schema": {
      "hrn": "<The new schema's HERE Resource Name>"
    }
   }
   

Mutable properties by layer type

Index layer

It is important to understand the ramifications of layer configuration updates you make after data has already been stored in a layer. For an index layer example, updating the schema association could adversely impact downstream consumer processes if those processes are not also updated to leverage the new schema. Also, only one schema can be associated with a layer so it is important that any schema change not only represent new data stored to the layer but also all pre-existing data stored in the layer. If not, consumers of the data will no longer be able to effectively read the data.

Furthermore, the schema configuration update will not be dynamically picked up by a running application or pipeline. Downstream pipelines will need to be restarted to pick up these changes.

Schema association

An update to the schema association is possible, but high risk. See the above index layer schema change example for more detail.

Digest

The digest field is immutable once set due to the severity of the data corruption possible with such a change. The server will ignore further attempts to update this configuration.

TTL

This configuration is mutable. Changes will take effect immediately once saved. Index layer TTL changes will be applied to all stored objects.

Versioned layer

Updates to a versioned layer will be in effect immediately upon server confirmation. Configuration changes to versioned layers are very limited to prevent data corruption and wider impacts across multiple dependent versioned layers.

Schema association

An update to the schema association is possible, but high risk. Only one schema can be associated with a layer and that schema must represent all data in the layer. Schema changes will break downstream processes and services until those downstream dependencies are also updated.

Digest

The digest field is immutable once set due to the severity of the data corruption possible with such a change. The server will ignore further attempts to update this configuration.

Stream layer

parallelization

Parallelization is implemented as Kafka topic partition count. It's immutable once set. To change it, the layer must be recreated.

dataInThroughputKbps

This configuration is mutable. Changes will take effect immediately once saved.

dataOutThroughputKbps

This configuration is mutable. Changes will take effect immediately once saved.

TTL

This configuration is mutable. Changes will take effect immediately once saved. Stream layer TTL changes will be applied to all stored objects.

ContentType & ContentEncoding

ContentType and Encoding configurations do not affect Kafka topic handling in any way. These configurations are mutable, enabling you to correct simple layer configuration mistakes after a layer is created.

If data is already saved in a stream layer before these changes are made, updates to these configurations will cause breaking changes to downstream consumers of the data. Data already saved in the layer prior to these changes will be saved using the old configuration settings. New data stored in the layer after these changes are saved using the new configuration settings so the overall data set will contain a mix. It is important that you realize the significant impact of these changes on your data before making them. Given the nature of stream layers (continuous feed of messages), these impacts are not permanent but you will need to wait for the messages encoded via the old layer configuration settings to be purged from the stream before restarting downstream pipelines and services.

Schema association

An update to the schema association is possible, but high risk. Only one schema can be associated with a layer and that schema must represent all data in the layer. Schema changes will break downstream processes and services until those downstream dependencies are also updated.

Interactive map layer

Searchable Properties

Searchable Properties is a list of strings, containing the properties you want to get indexed for faster searching. Searchable properties can be updated at any time.

If the layer encompasses less than 10,000 features, all properties are searchable.