Secret Store Supervisor Service Stuck in "Removing" State

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The Secret Store Supervisor (SSS) service on the Supervisor Cluster becomes stuck in the removing state and fails to clean up completely.
• In the vmware-system-appplatform-operator-system_vmware-system-psp-operator-mgr logs you see the message 

  FakeAttached PVC reconciling

Cause

This issue occurs when the vSphere CSI controller loses track of or hangs on reconciling the PVC/PV attachments associated with the Secret Store Supervisor service. The stuck attachment state prevents Kubernetes finalizers from releasing the storage resources, leaving the service in a perpetual removing state.

Resolution

1. Perform a rolling restart of the vSphere CSI controller deployment:
   Restart the CSI controller deployment in the Supervisor Cluster to reset active reconcilers and refresh the volume attachment state: 

   kubectl rollout restart deployment vsphere-csi-controller -n <csi-controller-ns>

2. Verify Persistent Volume (PV) state change:
   Check the status of the PV associated with the Secret Store Supervisor service. After restarting the CSI controller, the PV should transition to a Terminating state:

   kubectl get pv -n <secret-store-ns>

3. Delete the stuck Persistent Volume Claim (PVC):
   Manually delete the hanging PVC  to allow finalizers to complete and clear both the PVC and PV:

   kubectl delete pvc <pvc-name> -n <secret-store-namespace>

4. Confirm removal: Verify that the PVC, PV, and Secret Store Supervisor service have successfully cleared from the Supervisor Cluster. 

Additional Information

If any components remain after completing these steps, contact Broadcom Support for further assistance.