Error: tls: unknown certificate authority when configuring Supervisor Cluster

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

The Supervisor cluster remains in a "Configuring" state during lifecycle management of the Supervisor Service tkg.vsphere.vmware.com. This typically occurs after certificate rotations or environment updates where controller managers retain stale certificate volume mounts.

• Supervisor status is stuck in "Configuring" in the vSphere Client.
• capi-controller-manager pods in the svc-tkg-domain-c## namespace are in Error or CrashLoopBackOff state.
• Pod logs show the following error: err="failed to discover extension \"runtime-extension\": http call failed: Post \"https://runtime-extension-webhook-service.svc-tkg-domain-c##.svc:443/...": remote error: tls: unknown certificate authority"

Environment

• vSphere Kubernetes Service (VKS)

Cause

The capi-controller-manager pods are unable to validate the CA chain of the runtime-extension-webhook-service. This is caused by pods retaining stale certificate data in their local volume mounts, resulting in a mismatch even if the underlying secrets are updated.

Resolution

Perform a rollout restart of the affected deployments to force the pods to refresh their certificate mounts.

1. Log in to the Supervisor Cluster using kubectl.
2. Restart the runtime extension controller manager: kubectl rollout restart deploy runtime-extension-controller-manager -n svc-tkg-domain-c##
3. Restart the Cluster API (CAPI) controller manager: kubectl rollout restart deploy capi-controller-manager -n svc-tkg-domain-c##
4. Verify the pods return to a Running (1/1) state: kubectl get pods -n svc-tkg-domain-c##