Ingestion Metrics

Ingestion Metrics is a default dashboard available to you in Grafana that shows the following metrics. For the list of Ingestion metrics, labels and their possible values refer to the Common Metrics and Labels for Ingestion section at the end of this chapter.

Messages Received By the HERE Platform Ingestion Endpoints

METRIC NAME DETAILS DESCRIPTION
Total number of all messages received from realm Metric incoming_requests_total, with labels data_type="single" and endpoint_name="ingest" Total number of all messages received (by HERE platform ingestion endpoints) from the organization / realm. Does not include messages having failed authentication or authorization or those not received due to network/other issues.
Total number of messages received Metric incoming_requests_total, with labels data_type="single" endpoint_name="ingest", catalog and layer, grouped by catalog and layer. Total number of messages received (by HERE platform ingestion endpoints) by Catalog by Layer. Does not include messages having failed authentication or authorization or those not received due to network/other issues.
Total size of all messages received from realm Metric raw_size_total, with labels data_type="single" and endpoint_name="ingest" Total size of all messages received from the organization / realm
Total size of messages received Metric raw_size_total, with labels data_type="single" endpoint_name="ingest", catalog and layer, grouped by catalog and layer. Total size of messages received by Catalog by Layer
Average message size received Metric raw_size_stats_mean, with labels data_type="single" endpoint_name="ingest", catalog and layer, grouped by catalog and layer. Average size of messages received by Catalog by Layer
Minimum message size received Metric raw_size_stats_min, with labels data_type="single" endpoint_name="ingest", catalog and layer, grouped by catalog and layer. Minimum size of messages received by Catalog by Layer
Maximum message size received Metric raw_size_stats_max, with labels data_type="single" endpoint_name="ingest", catalog and layer, grouped by catalog and layer. Maximum size of messages received by Catalog by Layer

Messages Successfully Ingested By the HERE Platform

METRIC NAME DETAILS DESCRIPTION
Total number of all messages ingested from realm Metric successful_requests_total, with labels data_type="single" and endpoint_name="ingest" Total number of all messages successfully ingested from the organization / realm.
Total number of messages ingested Metric successful_requests_total, with labels data_type="single", endpoint_name="ingest", aid, catalog and layer, grouped by catalog and layer. Total number of messages successfully ingested by Catalog by Layer
Total size of all messages ingested Metric raw_size_ingested_total, with labels data_type="single" and endpoint_name="ingest" Total size of all messages ingested from the organization / realm
Total size of messages ingested Metric raw_size_ingested_total, with labels data_type="single" endpoint_name="ingest", aid,catalog and layer, grouped by catalog and layer. Total size of messages ingested by Catalog by Layer
Average message size ingested Metric raw_size_stats_ingested_mean, with labels data_type="single", endpoint_name="ingest", catalog and layer, grouped by catalog and layer. Average size of messages ingested by Catalog by Layer
Minimum message size ingested Metric raw_size_stats_ingested_min, with labels data_type="single", endpoint_name="ingest", catalog and layer, grouped by catalog and layer. Minimum size of messages ingested by Catalog by Layer
Maximum message size ingested Metric raw_size_stats_ingested_max, with labels data_type="single", endpoint_name="ingest", catalog and layer, grouped by catalog and layer. Maximum size of messages ingested by Catalog by Layer
Total number of message ingestion failures Metric failed_requests_total, with labels data_type="single", endpoint_name="ingest", aid, catalog and layer, grouped by catalog and layer. Total number of message ingestion failures (messages received but not ingested) by Catalog by Layer (for example, 400’s and 500’s, but not failures due to authorization failures or nonexistent catalog destinations).

Message Rate / Throughput of Ingested Messages

METRIC NAME DETAILS DESCRIPTION
Number of bytes per second (Mean and Max) Metric raw_size_ingested_total, with labels data_type="single", endpoint_name="ingest", aid, catalog and layer, grouped by catalog and layer. Number of bytes ingested per second by Catalog by Layer
Number of messages per second (Mean and Max) Metric successful_requests_total, with labels data_type="single", endpoint_name="ingest", catalog and layer, grouped by catalog and layer. Number of messages ingested per second by Catalog by Layer

SDII Specific Data Ingestion Metrics by the HERE Platform SDII Ingestion Endpoint

METRIC NAME DETAILS DESCRIPTION
Number of messages received by the platform SDIIMessageList endpoint Metric incoming_requests_total, with labels data_type="multi", endpoint_name="ingest", aid, catalog and layer, grouped by catalog and layer. Number of SDII multi-messages received (by SDIIMessage endpoint) by Catalog by Layer
Number of messages ingested by the platform SDIIMessageList endpoint Metric successful_requests_total, with labels data_type="multi", endpoint_name="ingest", aid, catalog and layer, grouped by catalog and layer. Number of SDII multi-messages ingested (by SDIIMessage endpoint) by Catalog by Layer
Number of single messages contained within MessageLists sent to the platform SDIIMessageList endpoint Metric num_of_messages_total, with labels data_type="multi", endpoint_name="ingest", aid, catalog and layer, grouped by catalog and layer. SDII multi-messages or "MessageList" include one or more single SDII messages. This number is the sum of single messages included in those multi-messages by Catalog by Layer.

SDII Specific Data Ingestion Metrics by Sensor Data Ingestion Platform Ingestion Endpoints (legacy endpoints)

METRIC NAME DETAILS DESCRIPTION
Number of messages received by the sensorData endpoint Metric incoming_requests_total, with labels data_type="single", endpoint_name="sdip.*" and aid, grouped by catalog, layer and aid. Number of SDII messages received (by sensorData endpoint) by AppID.
Number of messages ingested by the sensorData endpoint Metric successful_requests_total, with labels data_type="single", endpoint_name="sdip.*" and aid, grouped by catalog, layer and aid. Number of SDII messages ingested (by sensorData endpoint) by AppID.
Number of message ingestion failures by the sensorData endpoint Metric failed_requests_total, with labels data_type="single", endpoint_name="sdip.*" and aid, grouped by catalog, layer and aid. Number of SDII message ingestion failures (not ingested) by AppID.
Number of messages received by the sensorDataMulti endpoint Metric incoming_requests_total, with labels data_type="multi", endpoint_name="sdip.*" and aid, grouped by catalog, layer and aid. Number of SDII messages received (by sensorDataMulti endpoint) by AppID.
Number of messages ingested by the sensorDataMulti endpoint Metric successful_requests_total, with labels data_type="single", endpoint_name="sdip.*" and aid, grouped by catalog, layer and aid. Number of SDII messages ingested (by sensorDataMulti endpoint) by AppID.
Number of single messages contained within MessageLists by sensorDataMulti endpoint Metric num_of_messages_total, with labels data_type="single", endpoint_name="sdip.*" and aid, grouped by catalog, layer and aid. SDII multi-messages or "MessageList" include one or more single SDII messages. This number is the sum of single messages included in those multi-messages by AppID.
Number of SDII message ingestion failures by the sensorDataMulti endpoint Metric failed_requests_total, with labels data_type="single", endpoint_name="sdip.*" and aid, grouped by catalog, layer and aid. Number of SDII message ingestion failures (not ingested) by AppID.

Common Metrics and Labels for Ingestion

Below the most common metrics for Ingestion.

  • incoming_requests_total
  • failed_requests_total
  • num_of_messages_total
  • successful_requests_total
  • raw_size_total
  • raw_size_stats_min
  • raw_size_stats_max
  • raw_size_stats_mean
  • raw_size_ingested_total
  • raw_size_stats_ingested_min
  • raw_size_stats_ingested_max
  • raw_size_stats_ingested_mean

All the Ingestion metrics can be filtered out with these labels:

LABEL NAME DESCRIPTION
aid to filter out for a particular application id.
catalog to filter out for a particular catalog id.
data_type to filter for incoming request with a single or multiple messages in the body, possible values are single or multi.
endpoint_name to filter out for a particular ingestion endpoint, possible values are ingestion or sdip.* (legacy endpoint). For the standard Ingestion service use the ingest value.
layer to filter out for the layer id.

results matching ""

    No results matching ""