TAS OTel Collector Update (tanzu-otel-collector-release v0.11.35) Breaks Metric Names for All Downstream Consumers
Article ID: 449948
Updated On:
Products
Issue/Introduction
After a foundation picks up tanzu-otel-collector-release v0.11.35 (stemcell/tile upgrade), any monitoring or observability system that consumes metrics from TAS's OTel Collector Prometheus exporter can begin showing missing, incomplete, or "no data" metrics for that foundation. This is not specific to any one product and affects every downstream consumer of that exporter endpoint (Tanzu Hub, Healthwatch, Wavefront, etc)
Foundations that have not picked up the updated collector release continue to report normally.
Environment
EAR v10.2.13
Platform Services v10.4.3
Tanzu Hub v10.4.3
tanzu-otel-collector-release v0.11.35, which moves the OpenTelemetry Collector from v0.153.0 to v0.155.0
Cause
tanzu-otel-collector-release v0.11.35 bumps the OpenTelemetry Collector from v0.153.0 to v0.155.0. Across v0.154.0/v0.155.0, the upstream Prometheus exporter introduced a breaking change to how metric-name suffixes are handled. The previously-used setting add_metric_suffixes: false is now deprecated and silently ignored, so unit suffixes get appended to exported metric names (e.g. a metric Hub expects as some_metric is now exported as some_metric_bytes or similar).
Because the exporter config is passed from the TAS tile down to the VMs as runtime config, every foundation running the updated collector release exports metrics under different names than Tanzu Hub's Platform View dashboard queries expect. The metrics are still being generated, scraped, and successfully delivered to Hub but they just no longer match the exact metric names the dashboards query for.
Resolution
tanzu-otel-collector 0.11.36 contains the fix for this issue and has been incorporated into the following published product versions:
TAS 10.2.14
TASW 10.2.14
TAS 10.4.4
TASW 10.4.4
The fix will also be in upcoming later release versions.
In lieu of upgrading, you can employ the following workaround:
In Ops Manager, go to the EAR tile > System Logging / OTEL Settings > "Open Telemetry Collector Configuration" form, and enter the following to override the deprecated hard-coded TAS runtime config:
config:
exporters:
prometheus/cf-hub-collector:
endpoint: ":65331"
translation_strategy: UnderscoreEscapingWithoutSuffixes
tls:
ca_pem: '{{ .hubcollector.ca }}'
cert_pem: '{{ .hubcollector.cert }}'
key_pem: '{{ .hubcollector.key }}'
min_version: "1.3"
service:
pipelines:
metrics:
exporters:
- prometheus/cf-hub-collector
After saving, apply changes to redeploy the EAR tile with the corrected exporter configuration. Dashboards should begin repopulating for the foundation once the new config is live and metrics are re-scraped under the corrected names.
Feedback
