Supervisor controlPlane Node NotReady Error "unable to load TLS certificates from existing bootstrap client config read from /etc/kubernetes/kubelet.conf: data does not contain any valid RSA or ECDSA certificates"

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• One of the three Supervisor Cluster control plane nodes is in a NotReady state, causing etcd to lose quorum. Containers on the affected node are in an exited state, and few pods are in a terminating state.

• Error from the supervisor tab in vCenter UI 

Cluster test is unhealthy:
Get "http://localhost:1080/external-cert/<supervior clone plane ip>/6443/version?timeout=2m0s": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

• Output of kubectl get nodes shows below

root@### [ ~ ]# kubectl get nodes
NAME     STATUS     ROLES                   AGE   VERSION
<node-1>  Ready     control-plane,master    571d  v1.25.6+vmware.wcp.2
<node-2>  NotReady  control-plane,master    571d  v1.25.6+vmware.wcp.2
<node-3>  Ready     control-plane,master    571d  v1.25.6+vmware.wcp.2

• SSH to the Supervisor node and check the kubelet log using command "journalctl -xeu kubelet"

unable to load TLS certificates from existing bootstrap client config read from /etc/kubernetes/kubelet.conf: data does not contain any valid RSA or ECDSA certificates

• The certificate file that was referenced  by symlink /var/lib/kubelet/pki/kubelet-client-current.pem was 0kb.

 

Environment

• VMware vSphere Kubernetes Service
• vSphere with Tanzu 8.x 
   

Cause

The certificate file that was referenced  by symlink /var/lib/kubelet/pki/kubelet-client-current.pem was 0kb.

Resolution

Replace certificates on the control plane using Replace vSphere Supervisor (Previously known as vSphere with Tanzu) Certificates

Verify the symlink referenced by /var/lib/kubelet/pki/kubelet-client-current.pem is no longer 0kb.