Unable to run kubectl commands after logging into Workload Cluster due to x509: Certificate Signed by Unknown Authority for kubernetes-extensions

Products

• VMware vSphere Kubernetes Service
• vSphere with Tanzu

Issue/Introduction

After successfully logging into the workload cluster through kubectl vsphere login and after running any kubectl command, an error message similar to the following is returned:

1109 memcache.go:265] couldn't get current server API group list: the server has asked for the client to provide credentials
error: You must be logged in to the server

 

When inspecting the kube-apiserver pod logs in the affected workload cluster, the following error message is present regarding kubernetes-extensions:

kubectl logs -n kube-system <kube-apiserver pod name>

webhook.go:154] Failed to make webhook authenticator request: Post "https://localhost:5443/tokenreview?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes-extensions")

invalid bearer token : tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes-extensions")

guest-cluster-auth-svc pod logs in the affected workload cluster show a tls bad certificate error at the same timestamp as previous kubectl vsphere logins and kubectl command failures:

kubectl logs -n vmware-system-auth <guest-cluster-auth-svc pod name>

tls: bad certificate

Environment

vSphere Supervisor

This issue can occur regardless of whether or not this workload cluster is managed by TMC.

Cause

Cert-manager service has generated an invalid certificate for the service responsible for authentication into workload clusters.

The authentication service listens on port 5443 and uses a certificate issued by CN kubernetes-extensions.

Noted cert-manager issue: https://github.com/cert-manager/cert-manager/issues/3495

Resolution

This issue has been addressed in vCenter 8.0 U3e.

The invalid certificate will need to be regenerated in the affected workload cluster.

This involves deletion and recreation of certificate and secret objects which can be destructive.

Please open a ticket to VMware by Broadcom Technical Support referencing this KB article.