Wavefront Node Collector Failing to collect metrics from cadvisor endpoint

Products

• DX OpenExplore

Issue/Introduction

Collector stops scraping cAdvisor metrics. 

time="<Date - Time>" level=info msg="Querying source" name="kubelet_summary:<ipAddress>:10250"
time="<Date - Time>" level=error msg="Error in scraping containers from 'kubelet_summary:<ipAddress>:10250': request failed - \"401 Unauthorized\", response: \"Unauthorized\""
time="<Date - Time>" level=info msg="Querying source" name="prometheus_source: https://<ipAddress>:10250/metrics/cadvisor"
time="<Date - Time>" level=error msg="Error in scraping containers from 'prometheus_source: https://<ipAddress>:10250/metrics/cadvisor': missing nodes/metrics permission in the collector's cluster role: error retrieving prometheus metrics from https://<ipAddress>:10250/metrics/cadvisor (http status 401 Unauthorized)"

Environment

Kubernetes, Amazon EKS, 

Cause

The BoundServiceAccountTokenVolume feature is enabled by default in Kubernetes versions to manage refreshing service account tokens This feature improves the security of service account tokens by allowing workloads running on Kubernetes to request JSON web tokens that are audience, time, and key bound.

Resolution

Since this service token refresh is handled at the Kubernetes level, we refer you to the documentation of your current version of Kubernetes for investigation and resolution of this issue.

Workaround: Restarting Damonset or Collector pod(s) temporarily resolve error by forcing the refresh of the Service Token.