The HERE platform allows you to archive stream data so that you can later query and process that data for non-real-time use cases. For example, if you want to run a batch process daily to find all pothole detection events recorded that day in the area surrounding a given city, you can use an index layer to index the pothole detection events by event time, event type, and location, and then archive the data. You can then query the data every 24 hours for pothole events in the area of the city as part of your batch process.
The following diagram illustrates the overall process of archiving stream data, then querying it:
The key points in the diagram are:
Data from the stream layer is archived by an application that you create and run as a platform pipeline. The archiving application uses the Data Archiving Library which is a Java library that reads data from a stream layer, aggregates it, and indexes it to the index layer. For more information, see:
The index layer contains the archived data and indexing attributes. It is a layer in a separate catalog from the stream layer. For more information, see:
Once data is archived, there are multiple ways of querying the data:
There are multiple ways to interact with the index layer.
To create an archiving solution for a stream layer, follow these steps.
If you do not already have a stream layer whose data you want to archive, create a stream layer. For instructions, see:
In a different catalog than the one that contains the stream layer you want to archive, create an index layer. For instructions, see: Creating a Layer
The archiving process is performed by an application you create and run in a pipeline. The easiest way to create an application is to start with one of the example applications included with the HERE Data SDK. These examples show how to use the Data Archiving Library to store data.
Creating an archiving application involves these steps:
See the README file included with the examples for more information.
The archiving pipeline must have
read access to the catalog containing the stream layer, and
write access to the catalog containing the index layer. Grant this access to the group ID under which the archiving pipeline will be created. For instructions on how to grant access, see:
To run the application, you must create a pipeline on the HERE platform. For instructions, see:
In the Portal, select the Pipelines tab and find your pipeline. It should be in the Running state.
Once the archiving pipeline is running and data has been archived to the index layer, you can query and obtain data using one of the following methods:
Make sure the app that's querying the index layer has
read permission to the index layer. For more information, see:
For information on parsing the data retrieved from index layer, see: