Pinniped Concierge stuck in a CrashLoopBackOff State on 9.x

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Pinniped concierge pods in the vmware-system-pinniped namespace are in a CrashLoopBackOff state.
• Pinniped pod logs show errors similar to:
  • failed to acquire lease vmware-system-pinniped/pinniped-concierge
  • lock is held by [pod-name] and has not yet expired
  • write attempt rejected as client is not leader

Cause

This issue is caused by a stale leader election lease lock in the vmware-system-pinniped namespace.

If the designated leader pod for Pinniped is abruptly terminated, the Kubernetes Lease object may remain assigned to the terminated pod without updating its renewTime. When new Pinniped pods start, they attempt to acquire the lock but are forced to wait for the lease duration. However, the liveness probes for these pods typically time out much earlier.

Because the pods cannot become the leader before the liveness probe kills them, they cannot generate valid TLS certificates or write to the API server, creating a crash loop deadlock. 

Resolution

Restore the Pinniped Service -
Identify and delete the stale lease object to allow a new leader election:

1. Log in to the Supervisor Cluster.
2. Check the status of the lease: 

   kubectl get lease pinniped-concierge -n vmware-system-pinniped -o yaml

3. Delete the stale lease object: 

   kubectl delete lease pinniped-concierge -n vmware-system-pinniped

4. Verify that the Pinniped pods recover and reach the Running state.