Orphaned container volumes cannot be deleted through vSphere Client

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• The associated vmdk files for the container volumes, pvc, pv has already been deleted.
• The container volumes are only showing on the vCenter UI : vCenter > datastore > Monitor > Cloud Native Storage > Container Volumes
• From the guest cluster or workload cluster, we will get an empty result if we run:
  
  k get pvc,pv -A | grep "pvc_name_showing_in_the_vCenter_UI"
  
  
• Restarting the csi-controller pod does not make any changes.
• We can find the pvc from from VCDB using sample volume id: hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh
  
  /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "SELECT * from vpx_storage_object_info where id='hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh'"

• The 'delete disk' column flag is set to 'false' - this is because it is unable to find the vmdk that is associated with it.
• From /var/log/vmware/vsan-health/vsanvcmgmtd.log we observe following events :
  
  info vsanvcmgmtd[######] [vSAN@#### sub=Workflow opID=xxx-#######-####] Workflow Delete volume suspended for vim.Task:task-######, due to async action StartDeleteVolume
Workflow Delete volume suspended
Delete a virtual storage object : The object or item referred to could not be found

Additional symptoms reported:

• OpenShift and vCenter integration using vSphere CSI Driver where PVC was created and later deleted from OpenShift.
• Volume no longer associated with any pod but still appears in vCenter.
• Error occurs when attempting to delete the volume object from vCenter.
• vCenter shows volume associated with a VMDK file that doesn't exist on the datastore.
• Container volumes become orphaned after deletion from Kubernetes/OpenShift clusters.

 

Environment

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Cause

Deleting the vmdk directly from the datastore int the /fcd directory is not the ideal way to delete the pvc or pv.

Since the associated vmdk files for the container volumes has already been deleted the container volumes has become stale/orphan.

 

Resolution

The ideal way to delete pvc and pv remains to be from the cli of respective guest cluster/workload cluster (in this case).

1. k get pvc,pv -n namespace_name
2. k delete pvc_complete_name -n namespace_name
3. k delete pv_complete_name -n namespace_name

If the vmdk has been mistakenly deleted, please refer the below steps :

1. Download & extract the govc binary in the vCenter's /tmp directory.
   • Download command: wget https://github.com/vmware/govmomi/releases/download/v0.32.0/govc_Linux_x86_64.tar.gz
2. Extract with this: tar -zxf govc_Linux_x86_64.tar.gz
3. Move the govc binary to the user directory : sudo mv govc /usr/local/bin/
4. Define environment variables to connect to VC :

export GOVC_URL=<vCenter_FQDN>
export GOVC_USERNAME=<[email protected]>
export GOVC_PASSWORD=<administrator_password>
export GOVC_INSECURE=true

5. Run this command to list the container volumes visible in the datastore :

govc volume.ls -dc=datacenter_name -ds=datastore_name 

6. Run this command to generate a csv file to get complete information of those container volumes

govc volume.ls -dc=datacenter_name -ds=datastore_name -json | jq -r '.[]' | jq '.[] | {cluster: .Metadata.ContainerCluster.ClusterId, pvc: .Name, namespace: .Metadata.EntityMetadata[0].Namespace, owner: .Metadata.ContainerCluster.VSphereUser, sizeGB: (.BackingObjectDetails.CapacityInMb/1024), datastoreUrl: .DatastoreUrl, id: .VolumeId.Id}' | jq -s '.' | jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv' > Datastore_name-cns-volumes.csv

7. Create a cns-vols-to-delete.txt file in the VCenter:

vi cns-vols-to-delete.txt

8. Inside the text file, update the volume ID of the container volume that we want to remove. (Output from step 6)
9. Run this remove command:

xargs -a cns-vols-to-delete.txt -I{} -d'\n' govc volume.rm {}

If the govc utility fails to delete it will give an output as : govc : object not found. The only option at this point is to reconcile the datastore inventory :

govc disk.ls -R -dc=datacenter_name -ds=datastore_name

Following reconciliation the stale or orphaned container volumes will get cleared from the vCenter.

 

Additional Information

Steps to delete container volume from vCenter MOB : 

1. Enable vSAN MOB on vCenter.
2. Navigate to https://<vcsa fqdns>/vsan/mob/?moid=cns%2dvolume%2dmanager&method=delete
3. Input the volume_id and set deleteDisk to true
4. Click Invoke Method
5. Then refresh the vCenter UI and check.