Pipeline Workflows

The HERE platform groups a user's work in the platform into a top-level entity, pipeline. Pipeline templates are provided to aid the user with creating pipelines. These templates contain reusable pipeline definitions - the implementation, as well as the input and output catalog schemas.

Within the Pipelines API, a version of the pipeline is a form of a pipeline and pipeline template.

For more information on these concepts, see the Pipelines API.

The OLP CLI provides tools for managing pipelines, such as:

  1. Prepare:
  2. Deploy:

For more details, see Pipeline Commands.

The figure below illustrates this workflow.

Pipeline Workflow
Pipeline Workflow

Create a Pipeline

First, you need to have a group ID. For more information see the Teams and Permissions User Guide.

Next, enter the pipeline create command.

olp pipeline create example-pipeline GROUP-848311d0-baef-4ec9-b499-cb8b88595515

The command creates a pipeline, associates it with your specified group and displays the pipeline ID. Note down this pipeline ID, as you need it later in the workflow.

Pipeline has been created.
ID: ec60ab85-a735-4dce-8413-b43cd5d5202a

Note

If you receive an error message about the specified group ID, check the Platform Profile page to verify that you have used the correct group ID. Also verify that your app is part of that group. If you or your app do not belong to a group, ask your team or organization's administrator to assign you or your app to a group.

Create a Pipeline Template

Follow the steps below to create a new pipeline template.

  1. Enter the pipeline template create command. Specify the template name, runtime environment (stream or batch), package, main class, a group that the pipeline belongs to, and input catalog IDs that are expected in the pipeline version configuration.

    For an example, see Create Batch Pipelines.

Linux
Windows
olp pipeline template create "example-pipeline-template" batch-2.0.0 \
   example-pipeline.jar com.here.pipeline.PipelineProcessor \
   GROUP-848311d0-baef-4ec9-b499-cb8b88595515 --input-catalog-ids optimized-map sensor-data
olp pipeline template create "example-pipeline-template" batch-2.0.0 ^
   example-pipeline.jar com.here.pipeline.PipelineProcessor ^
   GROUP-848311d0-baef-4ec9-b499-cb8b88595515 --input-catalog-ids optimized-map sensor-data

The command creates a pipeline template and displays its ID. Note down this template ID, as you need it later in the workflow.

  1. To verify the pipeline template, enter the pipeline template show command.

    olp pipeline template show ac42c041-98ff-43d9-b0c3-0002fa756f49 --json
    

    The command displays the following results.

    {
       "created":"2018-03-30T12:40:39.577Z",
       "groupId":"GROUP-848311d0-baef-4ec9-b499-cb8b88595515",
       "defaultClusterConfiguration":{
          "supervisorUnits":1,
          "workerUnits":1,
          "workers":1
       },
       "name":"example-pipeline-template",
       "packageId":"ab8ea6b8-5a81-4070-9b84-ca4f28b67e22",
       "entryPointClassName":"com.here.pipeline.PipelineProcessor",
       "description":"",
       "id":"ac42c041-98ff-43d9-b0c3-0002fa756f49",
       "state":"creating",
       "defaultRuntimeConfiguration":"",
       "updated":"2018-03-30T12:40:39.577Z",
       "runtimeEnvironment":"batch-2.0.0",
       "inputCatalogIds": ["optimized-map", "sensor-data"]
    }
    

Create a Pipeline Version

Follow the steps below to create a new pipeline version.

  1. In the pipeline-config.conf file, specify the mapping from the input catalog's fixed identifier to catalog HRNs.

    Example pipeline-config.conf file:

    
       pipeline.config {
        output-catalog { hrn = "hrn:here:data:::example-output" }
        input-catalogs {
          optimized-map { hrn = "hrn:here:data:::here-optimized-map-for-location-library-1-1" },
          sensor-data   { hrn = "hrn:here:data:::here-example-vehicle-sensor-data-berlin" }
        }
       }
       

    Note: Fixed Identifiers

    Pipeline implementations may bind to and distinguish between multiple input catalogs via the fixed identifiers. Fixed identifiers are defined in a pipeline template. In contrast, HRNs are defined for each pipeline version, so that the same pipeline template may be reused in multiple setups.

  2. Enter the pipeline version create command and specify the version name, pipeline ID, pipeline template ID, and the path to your pipeline-config.conf file.

Linux
Windows
olp pipeline version create example-pipeline-version ec60ab85-a735-4dce-8413-b43cd5d5202a \
   ac42c041-98ff-43d9-b0c3-0002fa756f49 /user/data/pipeline-config.conf
olp pipeline version create example-pipeline-version ec60ab85-a735-4dce-8413-b43cd5d5202a ^
   ac42c041-98ff-43d9-b0c3-0002fa756f49 /user/data/pipeline-config.conf

For more information on pipeline configuration, see Pipeline Configuration.

Note down this version ID, as you need it later in the workflow.

  1. To verify your pipeline version, enter the pipeline version show command.

Linux
Windows
olp pipeline version show \
   ec60ab85-a735-4dce-8413-b43cd5d5202a 51e86a6c-a99c-450a-9e06-1b5609932ce9
olp pipeline version show ^
   ec60ab85-a735-4dce-8413-b43cd5d5202a 51e86a6c-a99c-450a-9e06-1b5609932ce9

The command displays the following results.


   Details of the olpclitestpipe pipeline:
   ID                       51e86a6c-a99c-450a-9e06-1b5609932ce9
   pipeline template ID     ac42c041-98ff-43d9-b0c3-0002fa756f49
   output catalog HRN       hrn:here:data:::example-output
   input catalogs           {"optimized-map":
                            "hrn:here:data:::here-optimized-map-for-location-library-1-1",
                            "sensor-data":
                            "hrn:here:data:::here-example-vehicle-sensor-data-berlin"}
   state                    ready
   created                  2018-04-13T13:00:45.909Z
   updated                  2018-04-13T13:01:18.346Z
   schedule                 none
   

Activate a Pipeline Version

Enter the pipeline version activate command and specify both, your pipeline ID and pipeline version ID.

Linux
Windows
olp pipeline version activate \
   ec60ab85-a735-4dce-8413-b43cd5d5202a 51e86a6c-a99c-450a-9e06-1b5609932ce9
olp pipeline version activate ^
   ec60ab85-a735-4dce-8413-b43cd5d5202a 51e86a6c-a99c-450a-9e06-1b5609932ce9

The command displays the following results.

Pipeline version has been activated.
Current state: scheduled

You can monitor the state of a pipeline version in one of the following ways:

A pipeline without scheduler should transition to running state soon after activation, and then to ready state when execution finishes. A pipeline with scheduler will wait in scheduled state for one of its input catalogs to change and then transition to running state. After execution it will transition back to scheduled state.

To get a full list of the available commands, enter olp --help.

results matching ""

    No results matching ""