Include the SDK in Your Project

Applications that run in the HERE Workspace may have dependencies on the following components:

  • HERE Data SDK for Java & Scala libraries, such as the Location Library, the Data Processing Library, or the Data Client Library.

  • Libraries provided as part of a runtime environment. The Workspace supports two runtime environments: batch and stream. The batch environment provides a minimal amount of libraries that Apache Spark applications use. The stream environment provides libraries that the Apache Flink applications use.

  • Protobuf schemas for encoding and decoding data, like schemas for HERE Map Content, HERE Weather or user provided ones

For all mentioned types of dependencies, the HERE Data SDK provides different ways for managing these dependencies.

HERE Data SDK Libraries

The libraries within the Data SDK have different versions. For example, Data Processing Library 3.1.13 and Data Client Library 0.2.24 are part of the same Data SDK release. Also, there are interdependencies within the Data SDK, for example the Location Library and the Data Processing Library depend on the Data Client Library, which you must take into account when managing dependencies.

To help with the dependency management, the Data SDK provides Bill Of Materials (BOMs) files. Each BOM is a Maven POM file that lists compatible versions of these libraries.

There are six different BOM files:

  • Scala 2.12:
    • sdk-batch-bom_2.12.pom - The file contains compatible versions of the HERE Data SDK for Java & Scala libraries, their dependencies and the libraries provided by the HERE Workspace Batch 3.0.0 runtime environment (Apache Spark 2.4.7). See SDK Libraries for Batch Processing with Scala 2.12.
    • sdk-stream-bom_2.12.pom - The file contains compatible versions of Data SDK libraries, their dependencies and the libraries provided by the HERE Workspace Stream 4.0 runtime environment (Apache Flink 1.10.3). See SDK Libraries for Stream Processing with Scala 2.12.
    • sdk-standalone-bom_2.12.pom - The file contains compatible libraries of Data SDK and their dependencies. Use this POM file for running the application on your own environment. See SDK Libraries for usage in Standalone mode with Scala 2.12.
  • Scala 2.11 (deprecated):

Warning: Deprecation Notice

Scala 2.11 is deprecated, use Scala 2.12 and SDK BOMs with the _2.12 suffix instead. Support for Scala 2.11 will be discontinued by the end of 2021.

For more information on migrating from Scala 2.11 to Scala 2.12, refer to the Scala Migration Guide.

Note that you need repository credentials for dependencies to resolve.

Include BOMs

You can include the BOM files in your project POM file in two different ways:

  1. As parent POM file (recommended):

    <!-- Inherit all properties, dependency management and plugin management from the SDK -->
<parent>
<groupId>com.here.platform</groupId>
<artifactId>sdk-batch-bom_2.12</artifactId>
<version>2.29.23</version>
</parent>

  2. Or, if your project already has a parent, as import dependency:

    <dependencyManagement>
<dependencies>
 <dependency>
    <!-- Import dependency management from the SDK, but properties and plugin management are skipped -->
    <groupId>com.here.platform</groupId>
    <artifactId>sdk-batch-bom_2.12</artifactId>
    <version>2.29.23</version>
    <type>pom</type>
    <scope>import</scope>
 </dependency>
</dependencies>
</dependencyManagement>

Note: Examples

All code snippets use sdk-batch-bom_2.12 as an example, but the same instructions are applicable to sdk-stream-bom_2.12 and sdk-standalone-bom_2.12. For Scala 2.11 the sdk-batch-bom, sdk-stream-bom, and sdk-standalone-bom artifactIds should be used. Be aware that only one SDK BOM file should be used in a project.

We recommend including the BOM files as parent because the files contain a plugin management section and useful properties.

For example, the plugin management section of sdk-batch-bom_2.12 and sdk-stream-bom_2.12 contains the platform profile to create a fat JAR. You need to upload this fat JAR to the Workspace before you run the application in the cloud.

Another important part of the plugin management section and platform profile is maven-shade-plugin that shades protobuf-java 3+. The shading mechanism is required because of a conflict between the version of the protobuf-java library used in Apache Flink and Apache Spark, and the version of this library used by Data SDK libraries and Protobuf layer schemas. The Data SDK uses a newer version 3+, but both stream and batch environments provide protobuf 2+.

You can activate the platform profile as follows: mvn -Pplatform package or mvn --activate-profiles=platform package.

If for some reason you cannot include the files as parent into your POM, check the content of the files and copy properties and plugin management configuration that are relevant for your projects.

Include Dependencies

As soon as your project includes a BOM file, the project can reference a Data SDK library or dependency without specifying the actual version of the library.

<dependencies>
  <!-- Reference to pipeline runner library from Data Processing Library -->
  <dependency>
      <groupId>com.here.platform.data.processing</groupId>
      <artifactId>pipeline-runner-java_2.12</artifactId>
  </dependency>
</dependencies>

The approach simplifies migration from one Data SDK version to a newer one. Changing the version of the sdk-batch-bom_2.12 updates the versions of all Data SDK libraries and their dependencies, and the versions of the libraries provided by the batch environment.

This approach simplifies the migration from one Data SDK version to a more recent one. If you change the version of the sdk-batch-bom_2.12 in your project, your project will use updated versions of the following:

  • Data SDK libraries
  • dependencies of all Data SDK libraries
  • libraries provided by the batch environment

Sometimes an application that depends on a different version of a package defined in the BOM may face runtime exceptions such as the following:

  • ClassNotFoundException
  • NoSuchMethodError

These exceptions are raised when classes or methods are not found in the library provided in the runtime environment. There are two common cases:

  1. The runtime environment contains an older version of the library, and the application requires a newer one with new methods and classes.
  2. The runtime environment contains a newer version of the library, but the application uses removed deprecated methods or classes from an old version.

If the library is referenced in your project then it can be solved by using a corresponding version of the library from the BOM files. However, it can happen that the version of one of the transitive dependencies conflicts with the version provided in the runtime environment. In this case, you need to shade this transitive dependency. For an example of how to shade a library, see the platform profile in sdk-batch-bom_2.12 or sdk-stream-bom_2.12. This profile shades the protobuf-java package as mentioned earlier.

For more information about how to import dependencies and BOMs, see the Maven Dependency Management Guide.

Note: Archetypes

The Workspace-provided batch and streaming archetypes contain BOMs and have the necessary dependencies in their project POM files. Only the Maven build system supports these archetypes.

Pipelines Runtime Environment Libraries

The HERE Workspace provides batch and streaming pipelines to run location data-based applications. Each pipeline job is executed within a specific runtime environment. This environment consists of the following:

  1. Java Runtime Environment (JRE)
  2. Apache Spark or Apache Flink framework

A batch (Spark) or streaming (Flink) environment comes with a list of packages that are loaded by the Java Virtual Machine (JVM) and take precedence over any other application-provided package.

To describe the list of packages that are part of the runtime environment, the Data SDK provides two environment BOMs:

  • Scala 2.12:
    • environment-batch-3.0.0.pom
    • environment-stream-4.0.0.pom
  • Scala 2.11 (deprecated):
    • environment-batch-2.1.0.pom
    • environment-stream-3.0.0.pom

Packages for a runtime environment are marked as provided in the corresponding environment BOM. For example, the Apache Spark package spark-core is marked as provided in environment-batch-3.0.0.pom:

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-core_${scala.compat.version}</artifactId>
  <version>${spark.version}</version>
  <scope>provided</scope>
</dependency>

Note that sdk-batch-bom_2.12 and sdk-stream-bom_2.12 files include the corresponding environment-batch-3.0.0.pom and environment-stream-4.0.0.pom files. A project that uses sdk-batch-bom_2.12 or sdk-stream-bom_2.12 doesn't need to explicitly include the corresponding environment POM file. For more information about the libraries provided by the pipeline runtime environments, see the corresponding environment POM files.

Protobuf Schemas

The HERE Workspace allows users to define their Protobuf schemas and distribute them in the Workspace. The schemas are stored in a special repository that requires platform credentials. You can access this repository only by means of the HERE Maven Wagon in Maven projects. You can access this repository only by means of the HERE Maven Wagon in Maven projects. For SBT projects, you can use the unofficial HERE SBT Resolver plugin.

Follow the instructions to install the Maven Wagon plugin or SBT Resolver plugin in README.md on GitHub.

If you have one of the plugins configured, then, to include a schema artifact shared with you in your project, follow the steps below.

  1. Go to the HERE portal.
  2. Find a shared schema you want to include and open Artifacts tab.
  3. For Maven projects, copy instructions and paste them in your project. For SBT projects, convert the instructions in SBT format like the following code and add them to build.sbt file:
libraryDependencies += "groupId" % "artifactId" % "version"

For more information, see Create and Extend Schemas.

Scala 2.11 to Scala 2.12 Migration Guide

This section describes high-level changes that have to be made to migrate your app from Scala 2.11 to Scala 2.12. Scala 2.12 support was introduced as part of the HERE Data SDK for Java & Scala 2.28 release.

  1. New SDK BOM files with the _2.12 suffix were added for each environment. To migrate to the Scala 2.12, update SDK BOM artifactId's to include _2.12 suffix as follows:

  2. Change your app's dependencies with the _2.11 suffix to use _2.12. It is recommended to use _${scala.compat.version} as a suffix. For instance, 

     <dependency>
   <groupId>org.apache.spark</groupId>
   <artifactId>spark-core_${scala.compat.version}</artifactId>
   <version>${spark.version}</version>
   <scope>provided</scope>
 </dependency>

    For convenience, all SDK BOM files declare the ${scala.compat.version} property. To use the property, SDK BOM must be added as a parent to your project.

  3. Schemas maintained by HERE have been updated to support Scala 2.12:

    • HERE Map Content Schema
    • SDII Schema
    • Sensoris Schema
    • Real-Time Traffic Schema
    • Weather Schema
    • Optimized Map for Location Library

    All Scala 2.12 bindings have the _2.12 suffix. To use the above-mentioned schemas, you have to add _2.12 to the schema artifactId.

    For more details on how to include a schema into the Maven or sbt project, refer to the schema page on the platform, see Sensoris schema as an example.

  4. Your own schemas published on the platform should be updated to support Scala 2.12. To do this, Scala 2.12 bindings should be added to the schema project.

    For more information on migrating schemas to use Scala 2.12, refer to the Schema Migration Guide.

  5. Since the 2.28 release, the HERE platform supports two additional pipeline environments for Scala 2.12. To run Scala 2.12 applications on the platform, use the newest versions as follows:

    For instance, if you use the OLP CLI, you have to change the <cluster type> parameter in the olp pipeline template create command.

Known Issues

  1. If your app uses maven-shade-plugin version 3.1.0 or older, you might want to upgrade it to the newer version. The recommended way to do it is to use the version from the SDK BOM.
  2. Spark's current Dataset API is broken for users of Java 8 lambdas or Scala 2.12. In this case you might want to add explicit type casting to MapFunction<> in you Java 8 code that uses Spark Dataset .map() method.

results matching ""

    No results matching ""