Tanzu Hub Red Metrics not displaying due to OpenTelemetry Collector configuration containing outdated metrics exporter

Products

• VMware Tanzu Platform Core
• VMware Tanzu Platform - Hub

Issue/Introduction

Operator recently upgraded Tanzu Hub to 10.4 from lower version.

Hub Dashboard reports "No Data" for many metrics (specifically RED metrics) under apps even they are showing with data under the App Metrics & Apps manager portal.

Environment

Product: Tanzu Hub 10.4

Cause

The OpenTelemetry Collector configuration changed between 10.4 and prior versions of Tanzu Hub. Compare 10.4 documentation  to 10.2 documentation for differences. The OpenTelemetry Collector configuration in Elastic Runtime should not contain "prometheus/cf-hub-collector" in metrics exporters block, otherwise it will cause problems with exporters service.

Resolution

The solution is to ensure OpenTelemetry Collector configuration matches 10.4 documentation and does not have any additional metrics exporters configured.

Perform steps:

1. Open the Elastic Application Runtime tile.
2. Go to Settings > System Logging/OTEL.
3. In the OpenTelemetry Collector Configuration field, validate that configuration has the following (remove prometheus/cf-hub-collector if present)
   

   exporters:
     otlp/app-collector-traces:
       endpoint: telegraf.hub-collector.service.internal:4317
       retry_on_failure:
         enabled: true
         max_interval: 90s
         max_elapsed_time: 600s
       balancer_name: round_robin
       tls:
         ca_pem: '{{ .cfhubcollector.ca }}'
         cert_pem: '{{ .cfhubcollector.cert }}'
         key_pem: '{{ .cfhubcollector.key }}'
   service:
     pipelines:
       traces:
         exporters:
           - otlp/app-collector-traces

After updating configuration, apply changes for Elastic Runtime tile and check afterwards that metrics display properly in Tanzu Hub dashboard.

If metrics are still not displaying after update then check the Otel-agent logs on Diego cells and Routers VM's for errors.