The HERE platform allows you to archive stream data so that you can later query and process that data for non-real-time use cases. For example, if you want to run a batch process daily to find all pothole detection events recorded that day in the area surrounding a given city, you can use an index layer to index the pothole detection events by event time, event type, and location, and then archive the data. You can then query the data every 24 hours for pothole events in the area of the city as part of your batch process.
The following diagram illustrates the overall process of archiving stream data, then querying it:
The key points in the diagram are:
Data from the stream layer is archived by an application that you create and run as a platform pipeline. The archiving application uses the Data Archiving Library which is a Java library that reads data from a stream layer, aggregates it, and indexes it to the index layer. For more information, see:
The index layer contains the archived data and indexing attributes. It is a layer in a separate catalog from the stream layer. For more information, see:
Once data is archived, there are multiple ways of querying the data:
- The Data Client Library provides Java/Scala libraries for reading data from index layers.
- The Command Line Interface allows you to read data from an index layer from a command line or script.
- The REST APIs Index and Blob can be used together to query and read the indexed data. The Index API returns the data handles for the data that matches your query. For example, if you query for events from a specific time frame and location, the response will contain the data handles for those events. Once you have the data handles that match your query criteria, you can use them to get the corresponding data using the Blob API.
There are multiple ways to interact with the index layer.
- Data Archiving Library: Use the Data Archiving Library to develop a custom application in Java which can run in a pipeline. Using the Data Archiving Library is the recommended way to store stream layer messages into an index layer. With the Data Archiving Library, you only need to implement the library's user-defined functions in your application to extract the indexing attributes for each message. Once you have your application created, you can package it then run the application in a pipeline. Note that the Data Archiving Library is only for writing to an index layer. You cannot use it to query data.
- Data Client Library: The Data Client Library provides Java and Scala APIs that you can use to interact with index layer. If the Data Archiving Library does not satisfy your requirements and you want to develop a custom application, then the Data Client Library is the recommended way to work with an index layer.
- REST API: Use the REST API if you want to create an application with a language that the Data Client Library does not support. You can use the REST API to interact with index layer.
- Command Line Interface: Use the command line interface (CLI) to work with an index layer from a command line or script.
To create an archiving solution for a stream layer, follow these steps.
If you do not already have a stream layer whose data you want to archive, create a stream layer. For instructions, see:
In a different catalog than the one that contains the stream layer you want to archive, create an index layer. For instructions, see: Creating a Layer
The archiving process is performed by an application you create and run in a pipeline. The easiest way to create an application is to start with one of the example applications included with the HERE Data SDK. These examples show how to use the Data Archiving Library to store data.
Creating an archiving application involves these steps:
- Implement the user-defined functions provided in the Data Archiving Library.
- Configure the
- Package the application into a fat JAR file.
See the README file included with the examples for more information.
The archiving pipeline must have
read access to the catalog containing the stream layer, and
write access to the catalog containing the index layer. Grant this access to the group ID under which the archiving pipeline will be created. For instructions on how to grant access, see:
To run the application, you must create a pipeline on the HERE platform. For instructions, see:
In the Portal, select the Pipelines tab and find your pipeline. It should be in the Running state.
Once the archiving pipeline is running and data has been archived to the index layer, you can query and obtain data using one of the following methods:
Make sure the app that's querying the index layer has
read permission to the index layer. For more information, see:
For information on parsing the data retrieved from index layer, see: