Error: ReconcileFailed state on Supervisor Control Plane VMs

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Supervisor Cluster remains in configuring state with the below error:

Initialized vSphere resources Deployed Control Plane VMs Configured Control Plane VMs Configuration error (since DD/MM/YYYY, HH:MM:SS AM) System error occurred on Master node with identifier ################################. Details: Log forwarding sync update failed: Command '['/usr/bin/kubectl', '--kubeconfig', '/etc/kubernetes/admin.conf', 'get', 'configmap', 'fluentbit-config-system', '--namespace', 'vmware-system-logging', '--ignore-not-found=true', '-o', 'json']' returned non-zero exit status 1.. Configured Load Balancer fronting the kubernetes API Server Configured Core Supervisor Services Service: velero.vsphere.vmware.com. Reason: "ReconcileFailed". Message: the server is currently unable to handle the request (get packages.data.packaging.carvel.dev). Service: tkg.vsphere.vmware.com. Reason: "ReconcileFailed". Message: the server is currently unable to handle the request (get packages.data.packaging.carvel.dev).

• All three Supervisor Control Plane VMs fail to respond to kubectl commands, returning "Unauthorized" errors.

Environment

VMware vSphere Kubernetes Service

Cause

The internal Kubernetes client certificates stored within the admin.conf file on the Supervisor Control Plane VMs have expired which can be validated by running the below command in the Supervisor Control Plane VMs

cat /etc/kubernetes/admin.conf | grep client-certificate-data | awk '{print $2}' | base64 -d | openssl x509 -noout -dates

Resolution

Perform the following steps on each Supervisor Control Plane VM to renew the certificates:

1. Log into the Supervisor Control Plane VM via SSH.
   
   
2. Renew all internal Kubernetes certificates: 
   
   kubeadm certs renew all
   
   
3. Persist the renewed certificates in the encrypted store: 
   
   /usr/lib/vmware-wcp/hypercrypt.py --reencrypt
   
   
4. Force the control plane components to reload the new certificates by restarting the API server, controller-manager, scheduler, and etcd containers. Use the following commands to remove the containers (they will be automatically restarted by kubelet): 
   
   crictl rm -f $(crictl ps --label io.kubernetes.container.name=kube-controller-manager -q) 
crictl rm -f $(crictl ps --label io.kubernetes.container.name=kube-scheduler -q) 
crictl rm -f $(crictl ps --label io.kubernetes.container.name=etcd -q) 
crictl rm -f $(crictl ps --label io.kubernetes.container.name=kube-apiserver -q)

 

Additional Information

For more information on vSphere Supervisor certificate management, see Replace vSphere Supervisor (Previously known as vSphere with Tanzu) Certificates.