The Data Archiving Library assists with archiving messages ingested via a stream layer. This library provides a data agnostic workflow that you can customize to produce and archive stream data in different formats including Protobuf, Avro and Parquet. The Data Archiving Library is included in the HERE Data SDK.
Along with this library, we provide examples showing how to archive SDII messages in each of these data formats and how to index your data by ingestion time, area (HERE Tile) and event type. You may alter these examples to index your data by a different set of attributes. Together, the Data Archiving Library and the examples provide you with an archiving pipeline solution that you can use to store your stream data for efficient querying customized to your requirements.
The use of the Data Archiving Library requires a basic understanding of some concepts and services related to the HERE platform: