Layers

The HERE platform supports the following types of data layers:

• Versioned
• Volatile
• Index
• Stream
• Object store
• Interactive map

Versioned, volatile, index, and stream layers are a set of partitions of a specific data type, functional property, and structure. You can use layers to segment data based on semantics. For example, in a catalog you can have one layer for road signs and another layer for road topology. You can also use layers to segment data based on schema. Layers can be overlaid spatially to construct a complete digital map.

Object store layers are a generic key/value store which does not enforce a data structure.

Versioned layers

A versioned layer stores slowly-changing data that must remain logically consistent with other layers in the catalog. When you want to update a catalog of versioned layers, all the layers related to the update (and partitions within a layer) must be updated in one publication so that they can be versioned together. For example, the HERE Map Content catalog contains several versioned layers, including Topology (road topology), Road (road attributes), and Place (points of interest). In each version of the catalog, these layers represent a consistent view of the world at that point in time. If a new road is built and there are new buildings containing new businesses along the new road, all three layers would need to be updated together in one publication so that in the new version of the catalog the Topology layer contains the new road, the Road layer contains the attributes for the new road, and the Place layer contains the names of the new businesses. If only the Place layer were to be updated with the new businesses, the layers would no longer represent a consistent view of the world because the Topology and Road layers would be missing the new road.

To achieve consistency between layers, any update that affects multiple layers must be published together in a publication. Updating multiple partitions of a versioned layer also happens in a publication to preserve the consistency and integrity of intra-layer and inter-layer references. A new catalog version is available only when all layers have been updated and the publication has been finalized.

You can access data as it existed at different points in time by referencing the version you want. Once a version has been published, the data in that version cannot be changed and can be removed only by removing the whole catalog version. Data within a version is immutable and consistent.

How data is versioned

The initial version of a catalog, before any data has been published to it, is -1. When data in a versioned layer is updated:

• The catalog version is incremented by one
• All layers and partitions that are updated in the publication have their versions updated to match the new catalog version
• All layers and partitions that are not updated retain their existing version number

It is important to note that only those layers and partitions that are updated have their version updated to the catalog's new version number. So, the version of a layer or partition represents the catalog version in which the layer or partition was last updated.

Requesting a version

When you request a particular version of data from a versioned layer, the partition that gets returned may have a lower version number than you requested. The following example illustrates this concept:

The red arrows show requests for data from particular versions. The dots along the line represent changes to a partition over time. The partition has been updated at catalog versions 1, 12, 24, 27, 35, 42, and 48. The current catalog version is 56.

The table below shows which partition version is returned for requests to different catalog versions.

Volatile layers

A volatile layer is a key/value store where values for a given key can change and only the latest value is retrievable. As new data is published, old data is overwritten.

Volatile layers use in-memory storage. Storing data in memory helps reduce data access latency and provides applications with consistently high throughput.

Consider using a volatile layer when you don't need older versions of the data. For example, if you want to make the latest weather information available, you could use a volatile layer to store the latest observations. When new weather observations are written to the layer, the old one is overwritten so that only the latest observations are available for data consumers.

Another use for a volatile layer is as a cache for applications requiring fast response times and consistently high throughput. When running complex, time-consuming computations, it is valuable to cache the computation results for future use. Correctly caching values that are accessed frequently not only reduces the load on the rest of the components in the cloud but also helps speed up responses to other clients requesting the same data. For example, say a client application performs a complex query that requires fetching data from five different versioned layers, sorting the data, matching the data, and running statistical analysis on the data to compute an optimized parameter. If it is likely that another client will request the same query, then the result of this complex query is a good candidate for caching in a volatile layer. This way, millions of clients can benefit from fast response times when submitting the same request. At the same time, the load on versioned layers and pipelines is significantly reduced.

Index layers

An index layer is part of an overall solution that enables you to index and store metadata and data in a way that is optimized for batch processing. The index layer itself is, as its name suggests, an index of the catalog’s data by attributes that you can later query. For example, if you want to run a batch process daily to find all pothole detection events recorded that day in the area surrounding a given city, you can use an index layer to index the pothole detection events by event time, event type, and location. You can then query the data every 24 hours for pothole events in the area of the city as part of your batch process. Index layers provide the flexible solution you need to easily store attributes of pothole events along with the location/time the event took place.

Like versioned layers, index layers are useful when you want to access historical data. The difference is that an index layer can be used when you do not need to maintain logical consistency across layer versions in the way versioned layers do. The other difference is that you can define your own attributes by which you want to index and query the data, whereas you cannot define your own attributes in versioned layers.

You can use an index layer in combination with a pipeline to append and get late events by event time. This ability to properly handle late events is important when your end user devices are online and offline at different times and where batches of data sent can include events with varying timestamps that need to be indexed appropriately with other events already received. Note that an index layer can only be used as input to a batch pipeline, not a stream pipeline. An index layer can be written to by either a batch or stream pipeline.

Index layers work in combination with the Data Archiving Library, which is available in the SDK. For more information about the Data Archiving Library, see:

Data Archiving Library Developer's Guide.

Stream layers

A stream layer is a queue that streams data to data consumers in real time. Consumers read the data in the order it is added to the queue. Once a consumer reads the data, the data is no longer available to that consumer, but the data remains available to other consumers.

Stream layers can be configured with a retention time, or time-to-live (TTL) which results in unconsumed data being removed after a specified period of time.

An example use of a stream layer is to handle data from vehicle sensors.

Object store layers

An object store layer is an object / file system storage solution which enables you to store raw data objects in the HERE platform without any kind of HERE proprietary partitioning.

An object store layer is a distributed and highly durable key/value store with the additional capability of listing the keys. You can upload data by referencing a key. You can overwrite that data by uploading new data to the same key. You can also delete the data using the key.

Object Store layers work in combination with the Data Client Library and its Hadoop File System Support component, which is available in the SDK. For more information, see Hadoop FS Support.

Interactive map layers

Interactive map layers offer a set of unique capabilities, enabling you to store, retrieve, search for, analyze and modify data at a feature (e.g., a place) and feature property (e.g., the name of the place) level.

Each layer type has a different set of characteristics and is therefore optimized for a different set of geospatial data use cases. Other layers store data in large tiles optimized for big data processing. While you can retrieve all data in a tile, loading data from a tile is not optimized for quick visual interactions. With interactive map layers, data is stored in GeoJSON and can be retrieved dynamically at any zoom level. That said, the major difference of the interactive map layer is that it is optimized for the visualization, analysis, and modification of data on a map (i.e., GIS functions). It is the rich, interactive, data inspection and analysis tools, and the ability to modify data real time on a map that set the interactive map layer apart.

Key features of interactive map layers include:

• Creating and modifying maps manually or programmatically; edits are published real-time and require no additional interaction.
• Modifying data a granular feature and feature property level.
• Adding and removing points, lines, and polygons directly on a map.
• Ability to retrieve data in different tiling schemes.
• Exploring and retrieving data by feature ID, bounding box, spatial search, property search, and features contained within a tile.
• Searching for data by values of feature properties (e.g., speed limits, type of place, address, name, etc.).
• Integration with HERE Studio inside HERE Workspace and popular 3rd party renderers like QGIS, Tangram, Leaflet, DeckGL, and MapboxGL, enabling visualization, analysis, and map editing.
• Data sampling, making it possible to efficiently render an excerpt of a very large data set for visual reference and analysis.
• Clustering using hexbins or quadbins to produce rich, visual data representations.
• Integration with batch and stream pipelines to process data at scale in the cloud.

For an introductory overview of interactive map layers, see the following video.

Live examples using interactive map layers and their source code can be found in the GitHub repository "HERE Interactive Map Layers Examples".