VKS dashboards not loading metrics

Products

• VCF Operations

Issue/Introduction

• The VCF Operations Kubernetes VKS Cluster dashboard shows the cluster objects in the VKS Clusters widget, but no data is loaded in any of the other widgets.

• The Supervisor Proxy service has been installed using steps in Install the Supervisor Management Proxy Service

• Prometheus has been installed in the VKS cluster using steps in Install Prometheus with Alertmanager

• Telegraf has been installed in the VKS cluster using steps in Install Telegraf

• Monitoring of the pods and containers has been enabled using steps in Enabling Monitoring for VKS Clusters

• Reviewing the /storage/log/var/log/httpd-south/access_log file on the cloud proxy reveals many HTTP 406 results when the telegraf agents in the VKS clusters are reporting metrics

  "POST /arc/tkgs/metric HTTP/1.1" 406 - "-" "Telegraf/1.33.0 Go/1.23.2 X:boringcrypto"

• Logs for the supervisor-management-proxy pod in the VKS Supervisor also indicate HTTP 406 results when the telegraf agents in the VKS Clusters are reporting metrics 

  kubectl logs -n svc-supervisor-management-proxy-domain-##### supervisor-managament-proxy-#####
  
  ... "POST /arc/tkgs/metric HTTP/1.1" 406 - 2401252 0 18 3 "-" "Telegraf/unknown Go/1.24.7 X:boringcrypto" ...

   

Environment

VCF Operations 9.0.x

Cause

The cloud proxy is a member of a Collector Group that has High availability enabled for Application Monitoring. VIP enabled cloud proxies are not supported for monitoring of VKS in VCF Operations 9.0.x (see What is vSphere Supervisor Monitoring)

Resolution

1. Log in to VCF Operations and navigate to Administration > Cloud Proxies > Collector Groups.

2. Click the 3 dots next to the Collector Group that the VCF Operations cloud proxy is part of and select Edit.

3. Uncheck Application Monitoring - Activate for high availability and click Save.