Create Streaming Pipelines

For more information on your options for streaming pipelines, see SDK Workflows.

The HERE Data SDK offers the following archetypes.

• streaming-java-archetype
• streaming-scala-archetype

Generate New Streaming Pipeline Projects

You can use Maven archetypes either in the command line or in an IDE such as Eclipse or IntelliJ IDEA. The examples below are for the command line option.

Follow the steps below to generate a new streaming pipeline project.

1. Create a wrapper project.
2. Add a schema definition module.
3. Create a streaming pipeline project.
4. Build your project.

After you build your project, you can start development.

Create a Wrapper Project

To create a wrapper for your project from the public Maven pom-root archetype, run mvn archetype:generate with the following parameters.

mvn archetype:generate -DarchetypeGroupId=org.codehaus.mojo.archetypes \
                       -DarchetypeArtifactId=pom-root \
                       -DarchetypeVersion=1.1 \
                       -DgroupId=com.example \
                       -DartifactId=myproject \
                       -Dversion=1.0-SNAPSHOT \
                       -Dpackage=com.example.myproject

mvn archetype:generate -DarchetypeGroupId=org.codehaus.mojo.archetypes ^
                       -DarchetypeArtifactId=pom-root ^
                       -DarchetypeVersion=1.1 ^
                       -DgroupId=com.example ^
                       -DartifactId=myproject ^
                       -Dversion=1.0-SNAPSHOT ^
                       -Dpackage=com.example.myproject

The above example uses the following values.

• archetypeGroupId org.codehaus.mojo.archetypes - do not change
• archetypeArtifactId pom-root - do not change
• archetypeVersion 1.1 - do not change
• groupId com.example
• artifactId myproject
• version 1.0-SNAPSHOT
• package com.example.myproject

Set your own values for the relevant parameters for your project.

Add a Data Schema Definition Module

For input data, use a schema defined in the layer configuration for your input catalog. Streaming pipelines require one or more input layers and a single output layer. They are defined by data schemas. You can either use a schema from the platform or create your own. If you use an existing schema for your output layer, you can include the schema artifact in your pipeline project.

When a new layer is created, you can specify a schema. This schema defines the structure for the data in the layer. You can select an existing schema or define your own.

The Data SDK provides a Maven Archetype for creating new schemas and extending existing data schemas, see Create and Extend Schemas.

To include an existing schema artifact into your pipeline project, follow the steps below.

1. Go to the HERE platform portal.
2. Open the Data tab and search for the target layer.
3. Go to the Schema tab for the layer.
4. Copy the Maven dependencies and paste them into your pipeline project POM file.

For more information, see the Artifact Service section in Dependency Management.

Create Your Own Schema

You can define a data schema to write output data for your streaming pipeline.

The Data SDK provides a Maven Archetype for creating new schemas and extending existing data schemas. For more information, see Create and Extend Schemas.

To add a data schema definition model to your project, enter the following command in your project folder.

mvn archetype:generate \
                       -DarchetypeGroupId=com.here.platform.schema \
                       -DarchetypeArtifactId=project_archetype \
                       -DarchetypeVersion=2.1.24 \
                       -DgroupId=com.example.myproject \
                       -DartifactId=model1 \
                       -Dversion=1.0.0 \
                       -Dpackage=com.example.myproject.model1 \
                       -DmajorVersion=0

mvn archetype:generate -DarchetypeGroupId=com.here.platform.schema ^
                       -DarchetypeArtifactId=project_archetype ^
                       -DarchetypeVersion=2.1.24 ^
                       -DgroupId=com.example.myproject ^
                       -DartifactId=model1 ^
                       -Dversion=1.0.0 ^
                       -Dpackage=com.example.myproject.model1 ^
                       -DmajorVersion=0

The above example uses the following values.

• archetypeGroupId com.here.platform.schema - do not change
• archetypeArtifactId project_archetype - do not change
• archetypeVersion 2.1.24 - do not change
• groupId com.example.myproject
• artifactId model1
• version 1.0.0
• package com.example.myproject.model1
• majorVersion 0

The last parameter defines the major version of your data schema and is in the package name. This allows you to use multiple major schema versions simultaneously.

Set your own values for these parameters for your project.

You may create one or more data schema definition models for your project.

To use the schema with your streaming pipeline, publish the schema project and add the resulting schema artifacts as a dependency. For Java bindings, use the artifact from the project with _java in the title. For Scala bindings, use the artifact from the project with _scala in the title.

To publish schema artifacts to the local repository, enter the following command in your schema project folder.

mvn install

For more information about the schema publishing, see Create and Extend Schemas.

Create a Streaming Pipeline Java Project

Add a Java streaming pipeline to your project and instantiate the Java streaming pipeline archetype.

To create and instantiate a Java streaming pipeline project, enter the following command.

mvn archetype:generate -DarchetypeGroupId=com.here.platform \
                       -DarchetypeArtifactId=streaming-java-archetype \
                       -DarchetypeVersion=1.0.794 \
                       -DgroupId=com.example \
                       -DartifactId=myproject \
                       -Dversion=1.0-SNAPSHOT \
                       -Dpackage=com.example.myproject

mvn archetype:generate -DarchetypeGroupId=com.here.platform ^
                       -DarchetypeArtifactId=streaming-java-archetype ^
                       -DarchetypeVersion=1.0.794 ^
                       -DgroupId=com.example ^
                       -DartifactId=myproject ^
                       -Dversion=1.0-SNAPSHOT ^
                       -Dpackage=com.example.myproject

The above example uses the following values.

• archetypeGroupId com.here.platform - do not change
• archetypeArtifactId streaming-java-archetype
• archetypeVersion 1.0.794- do not change
• groupId com.example
• artifactId myproject
• version 1.0-SNAPSHOT
• package com.example.myproject

To use your schema in a pipeline project, you have to include the dependency in the project POM file. For Java bindings, use the artifact from the schema project with _java in the title.

Create a Streaming Pipeline Scala Project

Add a Scala streaming pipeline to your project and instantiate the Scala streaming pipeline archetype.

To create and instantiate a Scala streaming pipeline project, enter the following command.

mvn archetype:generate -DarchetypeGroupId=com.here.platform \
                       -DarchetypeArtifactId=streaming-scala-archetype \
                       -DarchetypeVersion=1.0.794 \
                       -DgroupId=com.example \
                       -DartifactId=myproject \
                       -Dversion=1.0-SNAPSHOT \
                       -Dpackage=com.example.myproject

mvn archetype:generate -DarchetypeGroupId=com.here.platform ^
                       -DarchetypeArtifactId=streaming-scala-archetype ^
                       -DarchetypeVersion=1.0.794 ^
                       -DgroupId=com.example ^
                       -DartifactId=myproject ^
                       -Dversion=1.0-SNAPSHOT ^
                       -Dpackage=com.example.myproject

The above example uses the following values.

• archetypeGroupId com.here.platform - do not change
• archetypeArtifactId streaming-scala-archetype
• archetypeVersion 1.0.794- do not change
• groupId com.example
• artifactId myproject
• version 1.0-SNAPSHOT
• package com.example.myproject

To use your schema in a pipeline project, you have to include the dependency in the project POM file. For Scala bindings, use the artifact from the schema project with _scala in the title.

Build Your Project to Run Locally

To build your project, enter the following command in your project folder.

mvn install

Build Your Project to Run on the Platform

To run your pipeline on the platform, you need to build a fat jar first. This can be done by running the following command.

mvn install -Pplatform