Federated Prometheus scrape job reports target as healthy but collects no metrics (TKGi cluster metrics missing on Healthwatch)

Products

• VMware Tanzu Platform - Cloud Foundry
• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware Tanzu Kubernetes Grid Integrated Edition (Core)
• VMware Tanzu Kubernetes Grid Integrated EditionStarter Pack (Core)
• VMware Tanzu Platform
• VMware Tanzu Platform Core
• VMware Tanzu Application Service

Issue/Introduction

A TKGi cluster located in one foundation was not having its metrics collected by the Healthwatch instance running in another foundation , despite the federated scrape job  being configured correctly against it.

Symptoms observed:

• Querying Prometheus/Healthwatch with any PromQL expression for the affected jobs (e.g. job=~"^jobname-.*") returns completely empty results, even with no time filter or across historic windows.
• The Prometheus /api/v1/targets endpoint shows the target as healthy (health: "up", no lastError), with a normal scrape duration (~20s) — ruling out network, firewall, or TLS issues.
• Running curl directly against the /federate endpoint (with the job's TLS certs and matchers) returns valid, non-empty metric payloads.

Environment

Tanzu Kubernetes Grid Integrated 

Tanzu Platform for Cloud Foundry 

 

Cause

The remote scrape target's system clock was drifting approximately 5 minutes behind the Prometheus/Healthwatch server.

By default, Prometheus scrape configs use honor_timestamps: true, which makes Prometheus trust the timestamp embedded in each metric sample from the target rather than using its own local scrape time. Because the target's clock was skewed, incoming sample timestamps fell outside the active window of Prometheus's TSDB head block, and the TSDB rejected them as "out of bounds." These samples were silently dropped during ingestion — before ever reaching memory or disk — which is why the target showed as healthy while queries returned nothing.

Resolution

Set honor_timestamps: false on the affected federated scrape job. This forces Prometheus to stamp incoming samples with its own local scrape time instead of the (skewed) timestamp reported by the remote target, allowing the TSDB to accept and store the data normally.

scrape_configs:
  - job_name: jobname
    scheme: https
    metrics_path: "/federate"
    scrape_interval: 60s
    scrape_timeout: 50s
    honor_labels: true
    honor_timestamps: false   # Prevents TSDB "out of bounds" drops caused by clock drift on the remote target

Steps to apply and verify:

1. Update the scrape job in prometheus.yml as shown above. (on the tsdb VM in Healthwatch deployment) 
2. Reload the Prometheus configuration with zero downtime: kill -HUP $(pgrep prometheus) 
3. Confirm the reload succeeded: the metric prometheus_config_last_reload_successful should equal 1.
4. Re-run a query against the previously-empty job, e.g. apiserver_request_duration_seconds_bucket{job=~"^jobname-.*"}, and confirm data now populates in real time.
5. As a permanent fix (not just a workaround), ensure NTP is correctly configured and synchronized on the affected target host — honor_timestamps: false masks clock drift rather than correcting it, and drift can still cause label/ordering issues elsewhere.
6. Also the same line should be added in the scrap config on the tile for persistent settings and apply changes should be done on the Healthwatch tile.