Bosh VM system metrics missing from OTEL Collector

Products

• VMware Tanzu Platform - Cloud Foundry

Issue/Introduction

Users with OpenTelemetry (otel) collector monitoring may find that BOSH-specific metrics, such as system_healthy or system_mem_percent, appear to be missing from the standard OTLP metric streams.

Environment

TAS/EAR with OTEL Collector

Cause

BOSH system metrics are emitted by the loggr-system-metrics-agent on each VM. While these are available locally on each VM, they are aggregated and forwarded to the OpenTelemetry collector specifically on the clock_global VM by the loggr-system-metric-scraper job.

Resolution

Verify loggr-system-metric-scraper and otel-collector are healthy on clock-global VM. Review the logs for these processes.

Ensure your downstream monitoring platform is configured to ingest metrics from the clock_global otel-collector.

Additional Information

Check the presence of the metrics in different components

system-metrics-agent

curl https://system-metrics:53035/metrics --cert /var/vcap/jobs/loggr-system-metrics-agent/config/certs/system_metrics_agent.crt --key /var/vcap/jobs/loggr-system-metrics-agent/config/certs/system_metrics_agent.key --cacert /var/vcap/jobs/loggr-system-metrics-agent/config/certs/system_metrics_agent_ca.crt --resolve system-metrics:53035:127.0.0.1 

otel-collector

curl -k https://otel-collector:65331/metrics --cert /var/vcap/jobs/otel-collector/config/certs/otel-collector.crt --key /var/vcap/jobs/otel-collector/config/certs/otel-collector.key --cacert /var/vcap/jobs/otel-collector/config/certs/otel-collector-ca.crt --resolve otel-collector:65331:127.0.0.1