Unable to run kubectl commands after logging into vSphere Kubernetes Cluster due to x509: Certificate Signed by Unknown Authority in Kube-Apiserver Logs

Products

• VMware vSphere with Tanzu
• vSphere with Tanzu

Issue/Introduction

After successfully logging into the vSphere Kubernetes cluster through kubectl vsphere login and after running any kubectl command, an error message similar to the following is returned:

• 1109 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
  
  error: You must be logged in to the server

 

When inspecting the kube-apiserver pod logs in the affected vSphere Kubernetes cluster, the following error message is present:

• kubectl logs -n kube-system <kube-apiserver pod name>

• webhook.go:154] Failed to make webhook authenticator request: Post "https://localhost:5443/tokenreview?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes-extensions")

Environment

vSphere 7.0 with Tanzu

vSphere 8.0 with Tanzu

This issue can occur regardless of whether or not this cluster is managed by TMC

 

Cause

Cert-manager service has generated an invalid certificate for the service responsible for authentication into vSphere Kubernetes clusters.

Resolution

This issue has been fixed the next release of 8.0. 

The invalid certificate will need to be regenerated in the affected vSphere Kubernetes cluster.

Please open a ticket to VMware by Broadcom Technical Support referencing this KB article.