vSphere Supervisor Metrics Fail to Populate Grafana Dashboards via Telegraf

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

Real-time vSphere Supervisor metrics processed by Telegraf agents fail to populate Grafana dashboards.

• Grafana dashboards remain empty or display no data.
• Telegraf pods show a completely healthy status in Kubernetes with zero runtime errors, connectivity drops, or warning logs.

Environment

• VMware vSphere with Tanzu (Supervisor Clusters)
• Telegraf Agent (deployed via ConfigMap/DaemonSet)
• InfluxDB (v1.x or v2.x)
• Grafana

Cause

This issue occurs when the Telegraf configuration is routed to send data directly to the Grafana Live API endpoint using the [[outputs.http]] plugin.

Architecturally, the Grafana Live API functions strictly as an ephemeral, real-time streaming channel for instantaneous user interface updates. It does not possess a persistent database storage backend to commit, index, or retain incoming data. Because the data structure sent by Telegraf is syntactically correct, the Grafana web server accepts the payload and Telegraf records completely clean logs, masking the fact that the ingested metrics are discarded immediately after streaming.

To display historical and persistent data on your dashboards, Telegraf must route metrics directly to a database backend (such as InfluxDB), which Grafana then queries.

Resolution

To resolve this issue, update the Telegraf configuration to send metrics directly to your InfluxDB database and restart the Telegraf pods to apply the changes.

Step 1: Update the Telegraf Configuration

1. Open your telegraf-config.yaml file.
2. Locate the [[outputs.http]] block at the bottom of the configuration.
3. Remove or comment out the block completely:

# [[outputs.http]]

#   url = "http://<GRAFANA_IP_OR_FQDN>:3000/api/live/push/telegraf"

#   data_format = "influx"

#   [outputs.http.headers]

#     Authorization = "Bearer <TOKEN>"

1. Replace the removed section with the appropriate InfluxDB output block below, depending on your InfluxDB database version:

For InfluxDB v1.x:

[[outputs.influxdb]]

  urls = ["http://<INFLUXDB_IP_OR_FQDN>:8086"]

  database = "telegraf"

  ## Uncomment below if authentication is required:

  # username = "telegraf_user"

  # password = "telegraf_password"

For InfluxDB v2.x:

[[outputs.influxdb_v2]]

  urls = ["http://<INFLUXDB_IP_OR_FQDN>:8086"]

  token = "<YOUR_INFLUXDB_API_TOKEN>"

  organization = "<YOUR_ORG_NAME>"

  bucket = "telegraf"

Step 2: Apply the Configuration Changes

Apply the updated ConfigMap to your Kubernetes cluster:

kubectl apply -f telegraf-config.yaml

Step 3: Restart the Telegraf Pods

Kubernetes does not automatically hot-reload running pod configurations when a volume-mounted ConfigMap is modified. To force the Telegraf agents to read the new InfluxDB database settings, you must restart the pods.

Run the following command to delete the active pods, forcing the DaemonSet controller to recreate them with the updated configuration:

kubectl delete pods -l name=telegraf -n vmware-system-monitoring

Once the newly spawned pods initialize, they will mount the updated configuration and stream data directly into the persistent InfluxDB datastore. Grafana will then be able to pull and display your metrics reliably.