Wavefront Node Collector Failing to collect metrics from cadvisor endpoint
search cancel

Wavefront Node Collector Failing to collect metrics from cadvisor endpoint

book

Article ID: 423926

calendar_today

Updated On:

Products

DX OpenExplore

Issue/Introduction

In Kubernetes environments using standard projected Service Account tokens, individual wavefront-kubernetes-collector pods may intermittently and permanently fail to scrape metrics from the local kubelet or cAdvisor endpoints (:10250 or /metrics/cadvisor).

Symptoms:

  • Log entries in the collector container show persistent 401 errors:
time="<Date - Time>" level=info msg="Querying source" name="kubelet_summary:<ipAddress>:10250"
time="<Date - Time>" level=error msg="Error in scraping containers from 'kubelet_summary:<ipAddress>:10250': request failed - \"401 Unauthorized\", response: \"Unauthorized\""
time="<Date - Time>" level=info msg="Querying source" name="prometheus_source: https://<ipAddress>:10250/metrics/cadvisor"
time="<Date - Time>" level=error msg="Error in scraping containers from 'prometheus_source: https://<ipAddress>:10250/metrics/cadvisor': missing nodes/metrics permission in the collector's cluster role: error retrieving prometheus metrics from https://<ipAddress>:10250/metrics/cadvisor (http status 401 Unauthorized)"

 

  • Once a collector pod hits this state, 100% of subsequent scrape attempts fail indefinitely (e.g., continuous failures over 24+ hours).

  • The pod never self-recovers.

  • Restarting the collector pod or DaemonSet temporarily resolves the issue until the next token expiration/rotation cycle.

  • Affected pods may hit this error within ~60–90 seconds of initial start or after a token rotation cycle.

 

Environment

Kubernetes, Amazon EKS, 

Cause

The root cause is a stale, cached HTTP Bearer Token in memory within the legacy wavefront-kubernetes-collector process.

  1. Token Rotation in Kubernetes: The BoundServiceAccountTokenVolume feature is enabled by default in Kubernetes versions to manage refreshing service account tokens This feature improves the security of service account tokens by allowing workloads running on Kubernetes to request JSON web tokens that are audience, time, and key bound.

  2. In-Memory Caching: Legacy versions of wavefront-kubernetes-collector (such as v1.13.0 and older DaemonSet setups) read the ServiceAccount token from disk into memory during HTTP client/transport initialization at process startup.

  3. Transport Stagnation: The HTTP client inside the collector process does not dynamic re-read or reload the updated token file from /var/run/secrets/... during ongoing scrape cycles. Once Kubernetes invalidates or rotates the token on disk, the collector continues transmitting the stale, expired token header over HTTP, resulting in persistent 401 Unauthorized rejections from the Kubelet API.

Resolution

Since this service token refresh is handled at the Kubernetes level, we refer you to the documentation of your current version of Kubernetes for investigation and resolution of this issue.

Workaround: Restarting Damonset or Collector pod(s) temporarily resolve error by forcing the refresh of the Service Token.

 

Additional Information

Frequently Asked Questions (FAQ)

Q1: Is wavefront-kubernetes-collector v1.13.0 expected to dynamic reload tokens from disk?

No. Version 1.13.0 and earlier legacy Go collector builds load the token string into memory at startup during HTTP client initialization. They lack dynamic disk-watching or middleware to reload projected tokens prior to individual HTTP scrape requests.

Q2: Is this bug patched in a newer minor version of wavefront-collector-for-kubernetes?

No. Active development on the legacy repository (wavefront-collector-for-kubernetes) has been discontinued. The project has been superseded by the Observability for Kubernetes Operator (observability-for-kubernetes). No backported patches for dynamic token reloading are available for legacy DaemonSet Helm chart versions (e.g., Helm chart wavefront-1.3.1).

Q3: Is migration to the Observability for Kubernetes Operator required to permanently resolve this issue?

Yes. The Observability for Kubernetes Operator natively supports modern Kubernetes ServiceAccount token projections, dynamic secret lifecycle management, and updated RBAC requirements. Migrating to the Operator is required to achieve a permanent fix rather than relying on pod-restart mitigations.

Q4: Are there configuration flags (e.g., HTTP idle timeout or keep-alive parameters) in v1.13.0 to force a token reload?

No. Idle connection or HTTP transport keep-alive settings do not trigger a disk read for the ServiceAccount token file. The token string remains cached in the client configuration object in memory regardless of connection pooling changes.

 

Powered by Wolken Software