Create Streaming Pipelines
For more information on your options for streaming pipelines, see SDK Workflows.
Note: Maven
Streaming archetypes are supported by the Maven build system only.
The HERE Data SDK offers the following archetypes.
streaming-java-archetype
streaming-scala-archetype
Generate New Streaming Pipeline Projects
You can use Maven archetypes either in the command line or in an IDE such as Eclipse or IntelliJ IDEA. The examples below are for the command line option.
Follow the steps below to generate a new streaming pipeline project.
- Create a wrapper project.
- Add a schema definition module.
- Create a streaming pipeline project.
- Build your project.
After you build your project, you can start development.
Create a Wrapper Project
To create a wrapper for your project from the public Maven pom-root
archetype, run mvn archetype:generate
with the following parameters.
mvn archetype:generate -DarchetypeGroupId=org.codehaus.mojo.archetypes \
-DarchetypeArtifactId=pom-root \
-DarchetypeVersion=1.1 \
-DgroupId=com.example \
-DartifactId=myproject \
-Dversion=1.0-SNAPSHOT \
-Dpackage=com.example.myproject
mvn archetype:generate -DarchetypeGroupId=org.codehaus.mojo.archetypes ^
-DarchetypeArtifactId=pom-root ^
-DarchetypeVersion=1.1 ^
-DgroupId=com.example ^
-DartifactId=myproject ^
-Dversion=1.0-SNAPSHOT ^
-Dpackage=com.example.myproject
The above example uses the following values.
- archetypeGroupId
org.codehaus.mojo.archetypes
- do not change - archetypeArtifactId
pom-root
- do not change - archetypeVersion
1.1
- do not change - groupId
com.example
- artifactId
myproject
- version
1.0-SNAPSHOT
- package
com.example.myproject
Set your own values for the relevant parameters for your project.
Add a Data Schema Definition Module
For input data, use a schema defined in the layer configuration for your input catalog. Streaming pipelines require one or more input layers and a single output layer. They are defined by data schemas. You can either use a schema from the platform or create your own. If you use an existing schema for your output layer, you can include the schema artifact in your pipeline project.
When a new layer is created, you can specify a schema. This schema defines the structure for the data in the layer. You can select an existing schema or define your own.
The Data SDK provides a Maven Archetype for creating new schemas and extending existing data schemas, see Create and Extend Schemas.
To include an existing schema artifact into your pipeline project, follow the steps below.
- Go to the HERE platform portal.
- Open the Data tab and search for the target layer.
- Go to the Schema tab for the layer.
- Copy the Maven dependencies and paste them into your pipeline project POM file.
For more information, see the Artifact Service section in Dependency Management.
Create Your Own Schema
You can define a data schema to write output data for your streaming pipeline.
The Data SDK provides a Maven Archetype for creating new schemas and extending existing data schemas. For more information, see Create and Extend Schemas.
To add a data schema definition model to your project, enter the following command in your project folder.
mvn archetype:generate \
-DarchetypeGroupId=com.here.platform.schema \
-DarchetypeArtifactId=project_archetype \
-DarchetypeVersion=2.0.0 \
-DgroupId=com.example.myproject \
-DartifactId=model1 \
-Dversion=1.0.0 \
-Dpackage=com.example.myproject.model1 \
-DmajorVersion=0
mvn archetype:generate -DarchetypeGroupId=com.here.platform.schema ^
-DarchetypeArtifactId=project_archetype ^
-DarchetypeVersion=2.0.0 ^
-DgroupId=com.example.myproject ^
-DartifactId=model1 ^
-Dversion=1.0.0 ^
-Dpackage=com.example.myproject.model1 ^
-DmajorVersion=0
The above example uses the following values.
- archetypeGroupId
com.here.platform.schema
- do not change - archetypeArtifactId
project_archetype
- do not change - archetypeVersion
2.0.0
- do not change - groupId
com.example.myproject
- artifactId
model1
- version
1.0.0
- package
com.example.myproject.model1
- majorVersion
0
The last parameter defines the major version of your data schema and is in the package name. This allows you to use multiple major schema versions simultaneously.
Set your own values for these parameters for your project.
You may create one or more data schema definition models for your project.
To use the schema with your streaming pipeline, publish the schema project and add the resulting schema artifacts as a dependency. For Java bindings, use the artifact from the project with _java
in the title. For Scala bindings, use the artifact from the project with _scala
in the title.
To publish schema artifacts to the local repository, enter the following command in your schema project folder.
mvn install
For more information about the schema publishing, see Create and Extend Schemas.
Create a Streaming Pipeline Java Project
Add a Java streaming pipeline to your project and instantiate the Java streaming pipeline archetype.
To create and instantiate a Java streaming pipeline project, enter the following command.
mvn archetype:generate -DarchetypeGroupId=com.here.platform \
-DarchetypeArtifactId=streaming-java-archetype \
-DarchetypeVersion=1.0.665 \
-DgroupId=com.example \
-DartifactId=myproject \
-Dversion=1.0-SNAPSHOT \
-Dpackage=com.example.myproject
mvn archetype:generate -DarchetypeGroupId=com.here.platform ^
-DarchetypeArtifactId=streaming-java-archetype ^
-DarchetypeVersion=1.0.665 ^
-DgroupId=com.example ^
-DartifactId=myproject ^
-Dversion=1.0-SNAPSHOT ^
-Dpackage=com.example.myproject
The above example uses the following values.
- archetypeGroupId
com.here.platform
- do not change - archetypeArtifactId
streaming-java-archetype
- archetypeVersion
1.0.665
- do not change - groupId
com.example
- artifactId
myproject
- version
1.0-SNAPSHOT
- package
com.example.myproject
To use your schema in a pipeline project, you have to include the dependency in the project POM file. For Java bindings, use the artifact from the schema project with _java
in the title.
Create a Streaming Pipeline Scala Project
Add a Scala streaming pipeline to your project and instantiate the Scala streaming pipeline archetype.
To create and instantiate a Scala streaming pipeline project, enter the following command.
mvn archetype:generate -DarchetypeGroupId=com.here.platform \
-DarchetypeArtifactId=streaming-scala-archetype \
-DarchetypeVersion=1.0.665 \
-DgroupId=com.example \
-DartifactId=myproject \
-Dversion=1.0-SNAPSHOT \
-Dpackage=com.example.myproject
mvn archetype:generate -DarchetypeGroupId=com.here.platform ^
-DarchetypeArtifactId=streaming-scala-archetype ^
-DarchetypeVersion=1.0.665 ^
-DgroupId=com.example ^
-DartifactId=myproject ^
-Dversion=1.0-SNAPSHOT ^
-Dpackage=com.example.myproject
The above example uses the following values.
- archetypeGroupId
com.here.platform
- do not change - archetypeArtifactId
streaming-scala-archetype
- archetypeVersion
1.0.665
- do not change - groupId
com.example
- artifactId
myproject
- version
1.0-SNAPSHOT
- package
com.example.myproject
To use your schema in a pipeline project, you have to include the dependency in the project POM file. For Scala bindings, use the artifact from the schema project with _scala
in the title.
Build Your Project to Run Locally
To build your project, enter the following command in your project folder.
mvn install
To run your pipeline on the platform, you need to build a fat jar
first. This can be done by running the following command.
mvn install -Pplatform