Data, Catalog, and Layer Metrics

Data, Catalog, and Layer Metrics is a default dashboard available to you in Grafana that shows the following metrics. The Technical Name column shows the metric name to use if you want to create a custom report. If the metric is calculated using a filter, then both the technical name and the filter are shown.

Data Metrics

METRIC DETAILS DESCRIPTION
Total Number of Catalogs catalog_total The number of existing catalogs created over the selected time period
Total Number of Layers layer_total The number of existing layers (all layer types) created across all catalogs over the selected time period
Total Number of Versioned Layers layer_total with layer_type="versioned" The number of versioned layers created across all catalogs over the selected time period
Total Number of Volatile Layers layer_total with layer_type="volatile" The number of volatile layers created across all catalogs over the selected time period
Total Number of Stream Layers layer_total with layer_type="stream" The number of stream layers created across all catalogs over the selected time period
Total Number of Index Layers layer_total with layer_type="index" The number of index layers created across all catalogs over the selected time period
Versioned Layers - Total Storage Volume (Usage) layer_used_bytes with layer_type="versioned" Sum of data volume across all versioned layers (all catalogs)
Volatile Layers - Total Capacity layer_allocated_bytes with layer_type="volatile" Sum of reserved capacity across all volatile layers (all catalogs)
Volatile Layers - Used Capacity layer_used_bytes with layer_type="volatile" Sum of used capacity across all volatile layers (all catalogs)
Stream Layers - Total Throughput Capacity in_throughput_mbps with layer_type="stream" + out_throughput_mbps with layer_type="stream" Sum of throughput capacity across all stream layers (all Catalogs)
Index Layers - Total Storage Volume (Usage) layer_used_index_blobstore_bytes with layer_type="index" Sum of data volume across all Index layers (all catalogs)

Catalog Metrics

METRIC DETAILS DESCRIPTION
Layer Count Per Catalog layer_total_by_catalog Total Number of Layers Per Catalog
Versioned Layer Storage Volume Per Catalog layer_used_bytes with layer_type="versioned" and catalog=~"$catalog" Total Data Volume of all Versioned Layers per Catalog
Volatile Layer Unique Capacity Per Catalog layer_unique_used_bytes with layer_type="volatile" and catalog=~"$catalog" Total Unique Capacity (Amount of Stored Data Without Replication) of all volatile layers per catalog
Index Layer Data Storage Volume Per Catalog layer_used_index_blobstore_bytes with layer_type="index" and catalog=~"$catalog" Total data volume of all Index layers per catalog

Volatile Layer Metrics

METRIC DETAILS DESCRIPTION
Capacity Allocated for Volatile Storage layer_allocated_bytes Total capacity configured
Capacity Used for Volatile Storage layer_used_bytes Total capacity used
Unique Capacity Used for Volatile Storage layer_unique_used_bytes Amount of stored data without replication
Data Redundancy for Volatile Storage layer_data_redundancy Storage redundancy set per volatile layer configuration.

Stream Layer Metrics

METRIC DETAILS DESCRIPTION
TTL per Stream Layer ttl_min Time-to-Live (TTL) - User configuration setting per Stream Layer
Throughput per Stream Layer in_throughput_mbps and out_throughput_mbps IN/OUT throughput (megabytes/second) per stream layer
Write Byte Rate Per Layer (MB/s) kafka_server_produce_byte_rate Byte rate per second for all messages written to a layer
Read Byte Rate Per Layer (MB/s) kafka_server_fetch_byte_rate Byte rate per second for all messages read from a layer
Messages written Per Layer stream_layer_messagesInRate Total number of messages written per stream layer. (For Dashboard UI displays use max() of this metric)

Versioned Layer Metrics

METRIC DETAILS DESCRIPTION
Durable Storage Used for Versioned Layer layer_used_bytes Total capacity used for versioned layer in the catalogs

Index Layer Metrics

METRIC DETAILS DESCRIPTION
TTL per Index Layer ttl_index_min Time-to-Live (TTL) - User configuration settings per index layer
Metadata Storage Used for Index Layers index_layer_metadata_storage_bytes Total volume of metadata storage per index layer

Common Metrics and Labels used for Catalog and Layers

The table below reports the most common metrics and their labels for Catalogs an Layers.

METRIC LABELS
catalog_total -
layer_total layer_type with values: index, stream, versioned, volatile
layer_total_by_catalog billingTag to filter out for a particular billing tag.
catalog to filter out for a particular catalog id.
catalogHrn to filter out for a particular catalog HRN
layer_used_bytes billingTag to filter out for a particular billing tag.
catalog to filter out for a particular catalog id.
catalogHrn to filter out for a particular catalog HRN.
clientAppId to filter out for a particular Client App Id.
layer to filter out for a particular layer id.
layer_type to filter out for a particular layer type.
project to filter out for a particular project id.
layer_unique_used_bytes billingTag to filter out for a particular billing tag.
catalog to filter out for a particular catalog id.
catalogHrn to filter out for a particular catalog HRN.
layer to filter out for a particular layer id.
layer_type to filter out for a particular layer type, in this label volatile is the only value.
layer_allocated_bytes billingTag to filter out for a particular billing tag.
catalog to filter out for a particular catalog id.
catalogHrn to filter out for a particular catalog HRN.
clientAppId to filter out for a particular Client App Id.
layer to filter out for a particular layer id.
layer_type to filter out for a particular layer type, in this label volatile is the only value.
project to filter out for a particular project id.
layer_used_index_blobstore_bytes billingTag to filter out for a particular billing tag.
catalog to filter out for a particular catalog id.
catalogHrn to filter out for a particular catalog HRN.
clientAppId to filter out for a particular Client App Id.
layer to filter out for a particular layer id.
layer_type to filter out for a particular layer type, in this label index is the only value.
project to filter out for a particular project id.
index_layer_metadata_storage_bytes billingTag to filter out for a particular billing tag.
catalog to filter out for a particular catalog id.
catalogHrn to filter out for a particular catalog HRN.
clientAppId to filter out for a particular Client App Id.
layer to filter out for a particular layer id.
layer_type to filter out for a particular layer type, in this label index is the only value.
project to filter out for a particular project id.
in_throughput_mbps and out_throughput_mbps billingTag to filter out for a particular billing tag.
catalog to filter out for a particular catalog id.
catalogHrn to filter out for a particular catalog HRN.
clientAppId to filter out for a particular Client App Id.
layer to filter out for a particular layer id.
layer_type to filter out for a particular layer type, in this label stream is the only value.
project to filter out for a particular project id.
kafka_server_fetch_byte_rate and kafka_server_produce_byte_rate clientId to filter out for a particular client id.
strCatalog to filter out for a particular catalog id.
strLayer to filter out for a particular catalog layer.
stream_layer_messagesInRate strCatalog to filter out for a particular catalog id.
strLayer to filter out for a particular catalog layer.
ttl_min billingTag to filter out for a particular billing tag.
catalog to filter out for a particular catalog id.
catalogHrn to filter out for a particular catalog HRN.
clientAppId to filter out for a particular Client App Id.
layer to filter out for a particular layer id.
layer_type to filter out for a particular layer type, in this label stream is the only value.
project to filter out for a particular project id.
ttl_index_min billingTag to filter out for a particular billing tag.
catalog to filter out for a particular catalog id.
catalogHrn to filter out for a particular catalog HRN.
clientAppId to filter out for a particular Client App Id.
layer to filter out for a particular layer id.
layer_type to filter out for a particular layer type, in this label index is the only value.
project to filter out for a particular project id.

results matching ""

    No results matching ""