Migrate Pipeline to New Run-Time Environment

Contents

• Migrate Batch-2.0.0 Pipeline to Batch-2.1.0 Environment (Apache Spark 2.4.2 with Support for History Server)
• Migrate Batch-2.1.0 Pipeline to Batch-3.0 Environment (Apache Spark 2.4.7)
• Migrate Stream-2.0.0 Pipeline to Stream-3.0.0 Environment (Apache Flink 1.10.1)
• Migrate Stream-3.0.0 Pipeline to Stream-4.0 Environment (Apache Flink 1.10.3)
• Migrate Stream-4.0 Pipeline to Stream-5.0 Environment (Apache Flink 1.13.5)

Migrate Batch-2.0.0 Pipeline to Batch-2.1.0 Environment (Apache Spark 2.4.2 with Support for History Server)

To migrate a Batch pipeline to Batch-2.1.0 environment (Apache Spark 2.4.2 with support for History Server), follow these steps:

1. Download the HERE platform SDK 2.12 or newer that contains the new sdk-batch-bom.pom environment file. For more information on using the new sdk-batch-bom.pom file, see the SDK Dependency Management.
2. Re-compile your pipeline code using the new sdk-batch-bom.pom file to create a new JAR file. For more information, see Batch Pipeline.
3. For an existing Batch pipeline, use the CLI or Web Portal to create a new Pipeline Template and a Pipeline Version with the batch-2.1.0 run-time environment and the new JAR file that was created in Step #2 above. From the Web Portal, you can save time by creating a copy of the existing Pipeline Version that uses a previous version of the batch environment and changing the run-time environment to batch-2.1.0, and using the new JAR file for the new Pipeline Version. For more information, see Pipeline Deployment.
4. Upgrade to the new Pipeline Version. For more information, see Upgrade via Portal and Upgrade via CLI.

For more information about pipelines general support for Apache Spark, see Batch Pipelines - Apache Spark Support FAQ.

↑ Top

Migrate Batch-2.1.0 Pipeline to Batch-3.0 Environment (Apache Spark 2.4.7)

There are no changes required from Pipeline runtime perspective. Please check the HERE Data SDK for Java and Scala documentation for changes with Batch-3.0

↑ Top

Migrate Stream-2.0.0 Pipeline to Stream-3.0.0 Environment (Apache Flink 1.10.1)

To migrate a Stream-2.0.0 pipeline to Stream-3.0.0 environment (Apache Flink 1.10.1), follow these steps:

1. Use the 2.17 version or newer of HERE Data SDK for Java and Scala that contains the new sdk-stream-bom.pom environment file. For more information on using the new sdk-stream-bom.pom file, see the SDK Dependency Management.
2. Re-compile your pipeline code using the new sdk-stream-bom.pom file to create a new JAR file. For more information, see Stream Pipeline.
3. For an existing Stream pipeline, use the CLI or Web Portal to create a new Pipeline Template and a Pipeline Version with the stream-3.0.0 run-time environment and the new JAR file that was created in Step #2 above. From the Web Portal, you can save time by creating a copy of the existing Pipeline Version that uses the stream-2.0.0 run-time environment and changing the run-time environment to stream-3.0.0, and using the new JAR file for the new Pipeline Version. For more information, see Pipeline Deployment.
4. Upgrade to the new Pipeline Version. For more information, see Upgrade via Portal and Upgrade via CLI.

Stream-3.0.0 Worker (Flink TaskManager) Memory Changes

Apache Flink 1.10 introduced improvements in the memory setup of TaskManagers. Stream-3.0.0 includes these improvements and provides pipeline users with granular control on the memory configuration of the TaskManagers. There are two categories of these changes in Stream-3.0.0 compared to Stream-2.0.0 as documented below:

• In Stream-2.0.0, the memory in a Worker unit is translated to be set as the taskmanager.heap.size. However, it includes not only the JVM heap but also other "off-heap" memory components. In Stream-3.0.0, the memory in a Worker unit is translated to taskmanager.memory.process.size, which is assigned to the TaskManager JVM process. Flink adjusts the rest of the memory components based on default values or additionally configured options (more on this below).
• In Stream-3.0.0, pipeline users can use all the memory configuration options via the Stream Configuration option. The memory configurations available to pipeline users include all the configurations listed within the official Flink documentation. It is important to ensure that the values set for these memory components are tuned in accordance with the Worker units configured for the TaskManager.

For more information on the memory improvements, configuration options and defaults, see the official Flink documentation below:

• Flink 1.10 - Memory Management Improvements
• Flink 1.10 - Detailed Memory Model
• Flink 1.10 - Memory Migration Guide

For more information about pipelines general support for Apache Flink, please see Stream Pipelines - Apache Flink Support FAQ.

Known Issues

Change in the Maven Modules of Table API (FLINK-11064)

Users that had a flink-table dependency before, need to update their dependencies to flink-table-planner and the correct dependency of flink-table-api-*, depending on whether Java or Scala is used: one of flink-table-api-java-bridge or flink-table-api-scala-bridge.

Change in Scala-Macro Generated Serializers (FLINK-11330)

Users that migrated to stream-3.0.0 and have the warning-related scalac options enabled (e.g. -Xfatal-warnings, -Ywarn-unused:*) might face the compilation errors with the following messages: local val in method createSerializer is never used, *.scala:1: Unused import. These messages are actually a result of the macro-expanded code inspection.

To prevent the compilation errors for Scala 2.11, you can choose one of the following options:

1. Using sbt, add the scalacOptions ~= (_.filterNot(_.startsWith("-Ywarn-unused"))) line to your build.sbt settings for the project.
2. Provided you use Maven, use the com.github.ghik:silencer-plugin_${scala.version} plugin as following:

<configuration>
  <compilerPlugins>
    <compilerPlugin>
      <groupId>com.github.ghik</groupId>
      <artifactId>silencer-plugin_${scala.version}</artifactId>
      <version>1.7.0</version>
    </compilerPlugin>
  </compilerPlugins>
  <args>
    <!-- Unused val in a Flink macro instantiation -->
    <arg>
      -P:silencer:globalFilters=local val in method createSerializer is never used
    </arg>
    <!-- Unused import in a Flink macro instantiation, that is reported on line '1' of the source file -->
    <arg>
      -P:silencer:lineContentFilters=^/\*$;^package;^//
    </arg>
  </args>
</configuration>

Environment Comparison

↑ Top

Migrate Stream-3.0.0 Pipeline to Stream-4.0 Environment (Apache Flink 1.10.3)

There are no changes required from Pipeline runtime perspective. Please check the HERE Data SDK for Java and Scala documentation for changes with Stream-4.0

↑ Top

Migrate Stream-4.0 Pipeline to Stream-5.0 Environment (Apache Flink 1.13.5)

To migrate a Stream-4.0 pipeline to Stream-5.0 environment (Apache Flink 1.13.5), follow these steps:

1. Use the 2.34 version or newer of HERE Data SDK for Java and Scala that contains the new sdk-stream-bom.pom environment file. For more information on using the new sdk-stream-bom.pom file, see the SDK Dependency Management.
2. Re-compile your pipeline code using the new sdk-stream-bom.pom file to create a new JAR file. For more information, see Stream Pipeline.
3. For an existing Stream pipeline, use the CLI or Web Portal to create a new Pipeline Template and a Pipeline Version with the stream-5.0 run-time environment and the new JAR file that was created in Step #2 above. From the Web Portal, you can save time by creating a copy of the existing Pipeline Version that uses the stream-4.0 run-time environment and changing the run-time environment to stream-5.0, and using the new JAR file for the new Pipeline Version. For more information, see Pipeline Deployment.
4. Upgrade to the new Pipeline Version. For more information, see Upgrade via Portal and Upgrade via CLI.

Stream-5.0 Supervisor (Flink JobManager) Memory Changes

Apache Flink 1.11 introduced improvements in the memory setup of the JobManagers. By moving from Flink 1.10 to 1.13, Stream-5.0 now includes these improvements and provides pipeline users with granular control on the memory configuration of the JobManager. There are two categories of these changes in Stream-5.0 compared to Stream-4.0 as documented below:

• In Stream-4.0, the memory in the Supervisor unit is translated to be set as the jobmanager.heap.size. However, it includes not only the JVM heap but also other "off-heap" memory components. In Stream-5.0, the memory in a Supervisor unit is translated to jobmanager.memory.process.size, which is assigned to the JobManager JVM process. Flink adjusts the rest of the memory components based on default values or additionally configured options (more on this below).
• In Stream-5.0, pipeline users can use all the memory configuration options via the Stream Configuration option. The memory configurations available to pipeline users include all the configurations listed within the official Flink documentation. It is important to ensure that the values set for these memory components are tuned in accordance with the Supervisor units configured for the JobManager.

For more information on the memory improvements, configuration options and defaults, see the official Flink documentation below:

• Flink 1.11 - Memory Management Improvements
• Flink 1.13 - Set up JobManager Memory
• Flink 1.13 - Migration Guide

For more information about pipelines general support for Apache Flink, please see Stream Pipelines - Apache Flink Support FAQ.

Known Issues

None

Environment Comparison

↑ Top

See Also

• Apache Spark 2.4.7
• Apache Spark 2.4.2
• Apache Flink 1.10.3
• Apache Flink 1.10.1
• Apache Flink 1.7.1
• Dependency Management
• Creating Pipelines
• Pipeline Deployment