Cleaning Up Orphaned PVCs Using govc After Kubernetes Cluster Decommissioning

Products

• VMware vCenter Server
• VMware vSphere Kubernetes Service

Issue/Introduction

• Numerous Persistent Volume Claims (PVCs) remain in the vCenter Database (VCDB) with corresponding VMDKs still present in the fcd folder.
• The VMs and nodes originally associated with these PVCs have been removed from the hosts and datastores.
• Attempting to delete these PVCs via the vSAN Managed Object Browser (MOB) reports success, yet the entries persist in the VCDB and the fcd folder.
• These orphaned PVCs continue to consume datastore capacity despite the absence of an active Kubernetes cluster.

Environment

vSphere with Tanzu

Security Services Platform

Cause

Orphaned VMDKs and PVC metadata left behind after K8s cluster deletion. Since the original supervisor or workload cluster no longer exists to manage the lifecycle, these volumes become "unmanaged" and must be manually purged from the datastore.

Resolution

Note: The below steps are to be performed only when there is no k8s cluster associated with the datastore and the datastore needs complete cleanup of orphaned pvcs.
If there exists a k8s cluster, the ideal way to delete pvc and pv remains to be from the cli of respective guest cluster/workload cluster.

k get pvc,pv -n namespace_name
k delete pvc_complete_name -n namespace_name
k delete pv_complete_name -n namespace_name

Steps to be followed for complete cleanup of the orphaned pvcs and vmdks from the datastore.

1. Download & extract the govc binary in the vCenter's /tmp directory.
   Download command: wget https://github.com/vmware/govmomi/releases/download/v0.32.0/govc_Linux_x86_64.tar.gz
2. Extract with this: tar -zxf govc_Linux_x86_64.tar.gz
3. Move the govc binary to the user directory : sudo mv govc /usr/local/bin/
4. Define environment variables to connect to VC :
   export GOVC_URL=<vCenter_FQDN>
export GOVC_USERNAME=<[email protected]>
export GOVC_PASSWORD=<administrator_password>
export GOVC_INSECURE=true
5. List all PVCs on the datastore and save them to a file cns-vols-to-delete.txt:
   govc disk.ls -dc="datacenter-id" -ds="datastore-name" | awk '{print $1}' > cns-vols-to-delete.txt
6. Delete each PVC listed in the file:
   cat cns-vols-to-delete1.txt | xargs -n 1 govc disk.rm -dc="datacenter-id" -ds="datastore-name"