Grafana show "no data" with all TKGI cluster metrics

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

After installing Healthwatch Exporter for TKGI , Grafana dashboard shows "No Data". In Healthwatch > tsdb > prometheus logs,  user permission errors are logged. 

ts=2024-08-19T14:02:09.804Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Pod: pods is forbidden: User \"healthwatch\" cannot list resource \"pods\" in API group \"\" at the cluster scope"
ts=2024-08-19T14:02:09.804Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: pods is forbidden: User \"healthwatch\" cannot list resource \"pods\" in API group \"\" at the cluster scope"

Environment

Healthwatch 2.x

Cause

Healthwatch has no permission to scape metrics from master node of TKGI clusters.

Resolution

Please refer to Healthwatch product document - Configuring TKGI Cluster Discovery, you will have to add Healthwatch role in TKGI plans so that Healthwatch can scrape metrics from TKGI clusters.