Pipeline Logging

By default, event and error log entries are sent to Splunk for review. How much information is reported in the logs depends on the logging level you select for each pipeline version when it is executed. Splunk is a data collection, indexing, and visualization engine for operational intelligence. For information on how to use Splunk, see the Splunk Enterprise User Documentation.

Pipeline Logging Basics

HERE platform pipelines use logging to provide more details during their operation. Different levels of logging are available for different purposes. HERE platform pipelines support the following levels of logging:

• Debug — Includes fine-grained informational events that are most useful to troubleshoot a pipeline.
• Info — Includes informational messages that highlight the progress of the pipeline at a coarse-grained level.
• Warn — Includes information on potentially harmful situations; including other run-time situations that are undesirable or unexpected, but not necessarily "wrong".
• Error — Includes other run-time errors or unexpected conditions such as error events that might still allow the pipeline to continue running.

Change the Logging Level

The Logging Level is set for a Pipeline Version. All the Jobs use the logging level associated with the corresponding Pipeline Version.

Logging Levels can be set at the root level for the entire pipeline and/or at the individual logger level for a pipeline class. Due to operational latency, it takes a few minutes for the changes to take effect. This may delay the availability of the logs at the new level in Splunk.

The logging level for a Pipeline Version can be changed only when it is in one of the following states:

• Running — When the logging-level is changed for a Pipeline Version in a Running state, the system will change the logging-level of the currently running job.
• Ready or Scheduled — When the logging-level is changed for a Pipeline Version in a Ready or Scheduled state, the system will run the future jobs using the new logging-level.
• Paused — For a Pipeline Version in the Paused state, when the logging-level is changed and Pipeline Version is resumed, the system will run the future jobs using the new logging-level.

API Solution

To change a Pipeline Version's log-level to DEBUG at the root-level, use the following command from the REST API:

PUT /v2/pipelines/{pipelineId}/versions/{versionId}/logging-configuration

with the following body:

{
 "configuration": {
 "loggers": {
  "root": {
  "level": "debug"
 }
  }
 }
}

For more information, see the Pipeline API Reference.

CLI Solution

To change a Pipeline Version's log-level to DEBUG at the root-level, use the following command from the OLP CLI:

olp pipeline version log level set <pipeline-id> <pipeline-version-id> [command parameters]

Example

To change the logging level of a Pipeline Version to DEBUG at the root-level:

olp pipeline version log level set f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --root DEBUG --json

On success, the command returns something like the following.

{"configuration": {"loggers": {
    "root": {"level": "debug"}
}}}

In this case, the pipeline confirms that the log-level has been set to debug.

For more information, see the CLI User Guide.

Retrieve the Logging Level

To check the current log-level configuration, you must retrieve the log-level setting for a specific Pipeline Version.

API Solution

To retrieve Current Logging Level via API, use the following command from the REST API:

GET /v2/pipelines/{pipelineId}/versions/{versionId}/logging-configuration

For more information, see the Pipeline API Reference.

CLI Solution

To retrieve Current Logging Level using the OLP CLI, use the following command:

olp pipeline version log level get <pipeline-id> <pipeline-version-id> [command parameters]

Example

To retrieve the current logging level.

olp pipeline version log level get f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79 --json

On success, this command returns something like the following:

{"configuration": {"loggers": {
    "logger": [
        {
            "level": "info",
            "name": "com.example.someLoggingClass"
        },
        {
            "level": "info",
            "name": "com.example.examplepkg.otherClass"
        }
    ],
    "root": {"level": "error"}
}}}

For more information, see the CLI User Guide.

Finding Pipeline Logs

Each running Pipeline Version has a unique URL where the logs for that pipeline are stored. You can access the logs from the portal by clicking on the See Log link on the display for the running pipeline, as shown in Figure 1.

When using the CLI, the log URL information is supplied by the pipeline whenever a pipeline version is activated, upgraded, or when a status request is submitted. However, if you are running your pipeline by interfacing directly with the Pipeline REST API, you will need to make a status request to discover the logging URL. For more information, see the Logs, Monitoring and Alerts User Guide .

Example

The CLI show function is one way you can check status. In the case of a specific job, you can use the following CLI command.

olp pipeline version job show f2fc50c4-a0ac-4c8a-9637-0d9b3a0d4a96 d77f288e-2c89-4c94-b4ba-79fbd1e26e79 e8e31070-4f4b-4e4b-a8e0-40425844cb75

Successful output returns:

{
    "catalogVersions": null,
    "created": "2018-03-01T15:23:04.618Z",
    "id": "e8e31070-4f4b-4e4b-a8e0-40425844cb75",
    "state": "failed",
    "updated": "2018-03-01T15:23:32.497Z",
    "loggingUrl": null
}

In this case, the loggingUrl parameter has a null value. This is what you would expect to find for a job that is not yet running. Once the job runs, the logging URL would be shown.

Custom Logging and Local Development

Your HERE platform pipelines may need to log additional custom events. You can add these in the pipeline application code, but there are certain limitations.

The HERE platform generally supports standard logging via SLF4J. If you require a specific logging back end that is SLF4J-compatible, you need to specify the configuration in the application.conf. Your runtime also requires an SLF4J backend. For more information on logging backend implementations for SLF4J, refer to the official SL4FJ Documentation.

• When a developer wants to add logging to their pipeline code, they should use the SLF4J abstracted log API. The HERE platform will provide a binding on the platform, and an API to control log-levels on the fly, as described above.
• No separate logging implementation JAR file should be included in the pipeline JAR file artifact; such as slf4j-api or slf4j-log4j12. For example, slf4j-api should be a provided JAR file defined in the BOM for the pipeline application's Fat JAR file.
• When doing local development and testing of pipeline code, the developer is free to provide any SLF4J binding they choose, although we recommend logback.

Tools

Event Logging is handled by an embedded version of Splunk. In general, you will not need to be concerned because Splunk runs in the background. However, you can create a new Splunk dashboard or run reports on the platform portal.

See Also

• Splunk Enterprise User Documentation
• Logs, Monitoring and Alerts User Guide
• Data Client Library Logging
• Pipeline API Reference
• Pipeline Monitoring
• Grafana User Documentation