Read interactive map layer data

The Data Client Library provides the class LayerDataFrameReader, a custom Spark DataFrameReader for creating DataFrames that contain the data for all supported layer type including interactive map layer.

Read process

The read operation works according to the following steps:

1. Spark connector analyzes your query and starts communication with the server in order to retrieve information for distributing your query across the spark cluster. Individual filters that are part of your query are taken into account at this stage already.
2. Spark will distribute your query across the workers in the cluster which will then start requesting their individual chunks of data from the server.
3. The data returned by the server is converted into a generic row format on the worker nodes.
4. The resulting rows are passed on to the Spark framework to return the finalized DataFrame.

Dataframe columns

Unlike other layer types, interactive map layers use a static row format when working with the Spark framework.

Data columns

A DataFrame for interactive map layers will contain the following columns:

Project dependencies

If you want to create an application that uses the HERE platform Spark Connector to read data from interactive map layer, add the required dependencies to your project as described in chapter Dependencies for Spark Connector.

Read interactive map layer data

The following snippet demonstrates how to access an interactive map layer based DataFrame from an interactive map layer of a catalog. Note that interactive map layer uses a dynamic schema. Therefore, you don't need to specify the format explicitly.

It is also possible to provide context for interactive map layer by passing an appropriate option. See the notes below for details on option name.

import com.here.platform.data.client.spark.InteractiveMapDataFrame.InteractiveMapContextOptionName
import com.here.platform.data.client.model.InteractiveMapContext.{DEFAULT, EXTENSION}
import com.here.platform.data.client.spark.LayerDataFrameReader.SparkSessionExt
import org.apache.spark.sql.SparkSession
val query = "mt_geometry=inboundingbox=(85, -85, 180, -180) and  p.row=contains=7"

log.info("Loading data from IML for default context.")
val readDF = sparkSession
  .readLayer(catalogHrn, layerId)
  .query(query)
  .load()

log.info("Data loaded for default context!")
val defaultContextCount = readDF.count()
log.info("Dataframe contains " + defaultContextCount.toString + " rows for default context.")

log.info("Loading data from IML for extension context.")
val readExtensionDF = sparkSession
  .readLayer(catalogHrn, layerId)
  .option(InteractiveMapContextOptionName, EXTENSION)
  .query(query)
  .load()

log.info("Data loaded for extension context!")
val extensionContextCount = readExtensionDF.count()
log.info(
  "Dataframe contains " + extensionContextCount.toString + " rows for extension context.")

import static com.here.platform.data.client.model.InteractiveMapContext.EXTENSION;
import static com.here.platform.data.client.spark.InteractiveMapDataFrameConstants.INTERACTIVE_MAP_CONTEXT_OPTION_NAME;

import com.here.hrn.HRN;
import com.here.platform.data.client.model.InteractiveMapContext;
import com.here.platform.data.client.spark.javadsl.JavaLayerDataFrameReader;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

JavaLayerDataFrameReader javaLayerDFReader = JavaLayerDataFrameReader.create(sparkSession);
String query = "mt_geometry=inboundingbox=(85, -85, 180, -180) and  p.row=contains=7";

Dataset<Row> inputDF = javaLayerDFReader.readLayer(catalogHrn, layerId).query(query).load();

// Show the schema and the contents of the DF
long defaultContextCount = inputDF.count();
log.info("Number of rows in dataframe with default context: " + defaultContextCount);

// When reading interactive map layer, one can additionally pass an optional context
Dataset<Row> interactiveDF =
    javaLayerDFReader
        .readLayer(catalogHrn, layerId)
        .option(INTERACTIVE_MAP_CONTEXT_OPTION_NAME, EXTENSION)
        .query(query)
        .load();

long extensionContextCount = interactiveDF.count();
log.info("Number of rows in dataframe with extension context: " + extensionContextCount);