Tanzu Guest Cluster Workloads Stuck at 0/1 Replicas with kube-controller-manager: Unauthorized Errors

Products

Tanzu Kubernetes Runtime

Issue/Introduction

Following a certificate renewal on vCenter or the Supervisor Cluster, workloads in a Tanzu Guest Cluster (TKC) fail to scale or reconcile. Deployments remain stuck at 0/1 READY replicas without spawning Pods or generating deployment events. Simultaneously, standard user SSO logins via kubectl vsphere login fail with Unauthorized errors.

To confirm that a cluster is experiencing this specific issue, verify the following log snippets and command outputs during initial triage:

1. Workload Deployment Status (`kubectl describe deploy`)

Deployments show 0/1 replicas and generate zero events because the control plane is not processing spec changes:

$ kubectl describe deploy <deployment-name> -n <namespace>
Name:                   <deployment-name>
Namespace:              <namespace>
Replicas:               1 desired | 0 updated | 0 total | 0 available | 1 unavailable
...
Events:                 <none>

2. Controller Manager Authorization Lockout (`kube-controller-manager` logs)

The kube-controller-manager pods across control plane nodes repeatedly fail leader election lock retrieval:

$ kubectl logs -n kube-system -l component=kube-controller-manager --tail=20
leaderelection.go:330] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
leaderelection.go:330] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized
leaderelection.go:330] error retrieving resource lock kube-system/kube-controller-manager: Unauthorized

3. Expired Control Plane Node Certificates (`kubeadm certs check-expiration`)

Running kubeadm directly on any guest control plane node shows that local client certificates have expired:

$ kubeadm certs check-expiration

4. Guest SSO Authentication Failures (`guest-cluster-auth-svc` logs)

Standard SSO user login commands fail:

$ kubectl vsphere login --vsphere-username [email protected] --server <supervisor-ip>
error: You must be logged in to the server (Unauthorized)

Inspecting the authentication daemonset pod logs in vmware-system-auth reveals token validation failures due to missing/desynchronized public keys:

$ kubectl logs -n vmware-system-auth -l app=guest-cluster-auth-svc 
1 server.go:120] Invalid token: failed to validate JWT

Environment

VMware vSphere Kubernetes Service

Cause

This issue is caused by two overlapping certificate and authentication failures across the infrastructure layers:

Expired Guest Control Plane Certificates (Primary Cause): The local client certificate stored on disk (/etc/kubernetes/controller-manager.conf) on the guest control plane nodes has expired. This prevents kube-controller-manager from authenticating to kube-apiserver to execute scaling and reconciliation loops.
vCenter Public Key Desynchronization (Secondary Cause): Following a vCenter certificate renewal, updated public keys were not automatically pushed down to the guest cluster's guest-cluster-auth-svc-public-keys ConfigMap, causing JWT validation failures for SSO logins.

Resolution

Step 1: Obtain Direct Administrative Access (`admin.conf`)

Bypass the failing SSO layer by extracting the administrative kubeconfig directly from the Supervisor Cluster:

kubectl get secret -n <supervisor-namespace> <cluster-name>-kubeconfig -o jsonpath='{.data.value}' | base64 -d > admin.conf

Step 2: Renew Certificates on Guest Control Plane Nodes

Execute these steps as root on all guest control plane nodes:

ssh vmware-system-user@<control-plane-node-ip>
sudo su -
kubeadm certs renew all

Step 3: Cycle Control Plane Static Pods

Force kubelet and running static pods to load the newly issued certificates into memory. Run as root on all guest control plane nodes:

systemctl restart kubelet
cd /etc/kubernetes/manifests/
mv *.yaml /root/ && sleep 15 && mv /root/*.yaml /etc/kubernetes/manifests/

Step 4: Synchronize Public Keys from vCenter

Force vCenter to push updated public keys down to the guest cluster:

SSH into the vCenter Server Appliance (VCSA) as root.
Restart the Workload Control Plane (wcp) service:

vmon-cli -r wcp

Wait 2 minutes for background key propagation.

Step 5: Restart Guest Authentication Pods

Restart the auth service on the guest cluster so it mounts the updated keys from vCenter:

kubectl --kubeconfig=admin.conf delete pods -n vmware-system-auth -l app=guest-cluster-auth-svc

Additional Information

Upon successful resolution, kube-controller-manager logs will transition from Unauthorized errors to active controller loop startup lines:

1 controllermanager.go:593] Started "csrapproving"
1 controllermanager.go:593] Started "replicationcontroller"
1 horizontal.go:168] Started HPA controller

Perform the following final verification checks:

# 1. Verify Controller Manager leadership lock
kubectl --kubeconfig=admin.conf logs -n kube-system -l component=kube-controller-manager --tail=20

# 2. Verify deployments scale to 1/1 READY
kubectl --kubeconfig=admin.conf get deploy -n <namespace>

# 3. Verify standard user SSO login
kubectl vsphere login --vsphere-username [email protected] --server <supervisor-ip> --insecure-skip-tls-verify --tanzu-kubernetes-cluster-namespace <supervisor-namespace> --tanzu-kubernetes-cluster-name <cluster-name>