Pipeline Status Definitions

OLP Pipeline Status is a default dashboard available to you in Grafana that shows the following metrics.

For more information on monitoring and troubleshooting pipelines, see Pipeline Monitoring in the Pipeline Developer Guide.

The following metrics report on the internal lifecycle of a batch or stream job. They do not necessarily correspond directly to the Pipeline Status as reported by the API. In addition, these metrics are only available while a pipeline job is in existence.

pipeline_jobs_submitted

This metric is reported with value 1 from the moment the first components of the pipeline cluster resources have started until the moment the job starts processing.

pipeline_jobs_running

This metric is reported with value 1 from the moment the job starts processing until the moment the job terminates. A job may terminate because of successful completion, failed completion, or if it is canceled or paused by the user.

pipeline_jobs_completed

This metric is reported with value 1 from the moment the job successfully completes processing until the moment the pipeline cluster resources have terminated.

pipeline_jobs_failed

This metric is reported with value 1 from the moment the job terminates with a failure until the moment the pipeline cluster resources have terminated.

pipeline_jobs_canceled

This metric is only applicable for Stream runtime environment and only reported in case the user issues the Pause or Upgrade request. This metric is reported with value 1 from the moment a savepoint is created and the job has stopped processing until the moment the pipeline cluster resources have terminated.

Note: Cancel Operation on Pipelines

The Cancel request is handled by the platform by simply terminating all cluster resources. It is a hard shutdown of the pipeline. The metrics will not show pipeline_jobs_canceled with a value of 1. The last that will have been reported is pipeline_jobs_running with a value of 1.

results matching ""

    No results matching ""