Metrics are not available under Healthwatch Dashboard in TKGI due to "tls: failed to verify certificate: x509: certificate signed by unknown authority"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition
• VMware Tanzu Kubernetes Grid Integrated Edition (Core)
• VMware Tanzu Kubernetes Grid Integrated Edition 1.x

Issue/Introduction

You will observe that Prometheus process is failing with an error as mentioned below under tsdb VM  :

tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"ca\")"

ts=2025-02-03T06:28:41.812Z caller=klog.go:108 level=warn component=k8s_client_runtime func=Warningf msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Pod: Get \"https://XYZ.net:8443/api/v1/pods?limit=500&resourceVersion=0\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"ca\")"

ts=2025-02-03T06:28:41.812Z caller=klog.go:116 level=error component=k8s_client_runtime func=ErrorDepth msg="pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Pod: failed to list *v1.Pod: Get \"https://XYZ.net:8443/api/v1/pods?limit=500&resourceVersion=0\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"ca\")"

 

Cause

From the docs under section - Configure the TKGI SLI Exporter VM , it is mentioned that "For CA certificate for TLS, provide the CA certificate."

If you provide a self-signed CA certificate, it must be the same CA that signs the certificate in the TKGI API.

NOTE : You may notice that "CA certificate for TLS" is different - TKGI API (CA signed) and TKGI SLI Exporter VM (self signed) . 

 

Resolution

Activate the Skip TLS certificate verification checkbox under "Configure the TKGI SLI Exporter VM" section in Healthwatch Exporter for TKGI tile and "TKGI cluster discovery" section in Health watch tile. Save it and "apply change".