Configuration File Reference

Contents

• pipeline-job.conf
• pipeline-config.conf
• pipeline template
• credentials.properties
• olpcli.ini
• application.properties

Overview

This topic is a summary of the various configuration files used in HERE platform pipelines. Most are only used in certain places, such as with the CLI, and are identified accordingly. In some instances, a configuration file may only be used in specific use cases, which are also described here.

pipeline-job.conf

CLI only

This file only applies to activating batch Pipeline Versions without the use of the scheduler (that is, in a Run Now mode).

Example content of pipeline-job.conf:

    pipeline.job.catalog-versions {
        output-catalog { base-version = 42 }
        input-catalogs {
            test-input-1 {
                processing-type = "no_changes"
                version = 19
                }
            test-input-2 {
                processing-type = "changes"
                since-version = 70
                version = 75
                }
            test-input-3 {
                processing-type = "reprocess"
                version = 314159
            }
        }
    }

Where:

• base-version of output catalog indicates the already existing version of the catalog on top of which new data should be published. This parameter will not be necessary in the future and will be removed. It is currently required to commit to the output catalog updated data.
• input-catalogs contains, for each input source, which is the most up-to-date version. This is the version that should be processed. In addition, information that specifies what has changed since the last time the job ran is also included (see the example above). Catalogs can be distinguished via the same identifiers present in the pipeline configuration file.

• processing-type describes what has changed in each input since the last successful run. Possible values include no_changes, changes, and reprocess.

  • no_changes indicates that an input catalog has not changed since the last run.
  • changes indicates that an input catalog has changed. A second parameter since-version is included to indicate which version of that catalog was processed during the last run.
  • reprocess does not specify whether an input catalog has changed or not. The pipeline is requested to reprocess that whole catalog instead of attempting any kind of incremental processing. This may be due to an explicit user request or to a system condition, such as the first time a pipeline runs.

↑ Top

pipeline-config.conf

CLI only

When you create a Pipeline Version using the CLI, the configuration of the new Pipeline Version is specified by the pipeline-config.conf file, the template file ID, and the pipeline ID. pipeline-config.conf is added to the classpath of the main user process.

Example content of pipeline-config.conf:

    pipeline.config {
         billing-tag = "test-billing-tag"
         output-catalog { hrn = "hrn:here:data:::example-output" }
         input-catalogs {
             test-input-1 { hrn = "hrn:here:data:::example1" }
             test-input-2 { hrn = "hrn:here:data:::example2" }
             test-input-3 { hrn = "hrn:here:data:::example3" }
         }
     }

Where:

• billing-tag specifies an optional tag to group billing entries for the pipeline.
• output-catalog specifies the HRN that identifies the output catalog of the pipeline.
• input-catalogs specifies one or more input catalogs of the pipeline: for each input catalog, its fixed identifier is provided together with the HRN of the actual catalog.

↑ Top

pipeline template

The pipeline template is an entity required by the Pipeline API for creating a Pipeline Version. From the platform portal, it is created by answering the questions in the UI. It can also be created using the pipeline template create CLI command. The parameters used in both cases include the following.

Required:

However, if you are using the API to create a template, the parameters are different. This is because the platform portal and CLI take care of uploading the pipeline JAR file to the Pipeline. When using the API, this is a separate step that creates a package based on the uploaded pipeline JAR file and assigns it a package ID. This is how all pipeline JAR files are identified and tracked.

Example

Results from pipeline template list CLI command:

{
    "pipelineTemplates": [
        {
            "created": "2018-03-01T15:16:53.796Z",
            "groupId": "GROUP-9479863e-a13b-4d35-9eb1-5a054669046e",
            "defaultClusterConfiguration": {
                "workerResourceProfileId": "HS1B",
                "supervisorResourceProfileId": "HS1B",
                "supervisorUnits": 1,
                "workerUnits": 1,
                "workers": 1
            },
            "name": "sparktestcompiler",
            "packageId": "68d723f6-2ae7-40e4-8c24-61512d511852",
            "entryPointClassName": "com.example.Main",
            "description": "",
            "id": "5c0660a3-0fb4-4f35-bcd0-be6ce25075f6",
            "state": "created",
            "defaultRuntimeConfiguration": "",
            "updated": "2018-03-01T15:16:53.796Z",
            "runtimeEnvironment": "batch-3.0"
        }
    ]
}

↑ Top

credentials.properties

Every application (such as a pipeline) functioning within the HERE platform must be registered with the HERE platform. This is done under the Apps and keys function in the platform portal. When registered, the credentials.properties file is created and must be used whenever loading that specific pipeline into the HERE platform. The following is an example of the credentials.properties file.

here.user.id = HERE-01966c94-aaf1-4ae2-a1y6-6516b3f9b6c1
here.client.id = mzLcb1rL8nskvDQpCAAO
here.access.key.id = BELUTk45QdaYGgZ9A_IMTA
here.access.key.secret = 108lI7w9m8G_6sIw9kng-PXGoeHQQ-cv6xByNOuMcRYixZZp...
here.token.endpoint.url = https://account.api.here.com/oauth2/token

For more information on how to use this file, see how to Get Your Credentials or the Identity & Access Management Guide.

↑ Top

olpcli.ini

CLI only

This file is created from the credentials.properties file. For additional information, see HERE Workspace for Java & Scala Developers.

↑ Top

application.properties

A custom configuration file (in Java Properties file format) made available on the running pipeline’s classpath. This file is constructed from the value of the Pipeline Template’s defaultRuntimeConfig property (API & CLI only) overridden on a key-by-key basis with the value of the Pipeline Version’s customRuntimeConfig property. The values of defaultRuntimeConfig and customRuntimeConfig are strings whose content represents a valid Java Properties file.

Example

    # Value of Pipeline Template's "defaultRuntimeConfig" property
    "myexample.threads = 3\nmyexample.language = \"en_US\"\nmyexample  .processing.window=300\nmyexample.processing.mode=stateless"

    # Value of Pipeline Version’s "customRuntimeConfig" property
    "myexample.threads=5\n\n myexample.processing.mode=    \"stateful\"\nmyexample.processing.filterInvalid = true"

    # The resulting Application.properties file on the pipeline classpath
    # (for the given values of "defaultRuntimeConfig" and "customRuntimeConfig")
    myexample.threads = 5
    myexample.language = "en_US"
    myexample.processing.window = 300
    myexample.processing.mode = "stateful"
    myexample.processing.filterInvalid = true

↑ Top

See Also

• Interfacing with the Pipeline Process