PVCs aren't deleted from the Supervisor when a VKS cluster is deleted

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server

Issue/Introduction

When you delete a VKS cluster without first deleting the PV/PVCs inside the guest cluster, the PVCs that were in the VKS cluster are retained on the Supervisor - even if the reclaim policy is set to "Delete" on the supervisor.

Environment

vCenter 7.x/8.x
VKS 3.x

Resolution

This is expected behaviour. The supervisor will retain PVCs from the VKS cluster that haven't been manually deleted. This allows VKS cluster recovery in case of accidental VKS cluster deletion.

In order to identify the PVCs that have been left hanging by previous VKS cluster deletions, you can use the following steps:

1. Identify UIDs of all active TKC clusters in the namespace. Example command: 
kubectl get cluster -n "$NAMESPACE" -o jsonpath='{.items[*].metadata.uid}

2. List all PVCs in the Supervisor Cluster namespace matching the uuid-uuid naming pattern. Example command: 
kubectl get pvc -n "$NAMESPACE" -o custom-columns=NAME:.metadata.name --no-headers | grep -E '^[0-9a-fA-F-]{36}-[0-9a-fA-F-]{36}$'

3. Extract the first UUID from each PVC name and compare it against the list of TKC UIDs.

4. Identify PVCs whose first UUID doesn't match any current TKC UID.

5. Delete these stale PVCs associated with non-existent or deleted TKC clusters

You can delete the PVCs through the UI or from the Supervisor kubectl context.