vCenter reports frequent failures of the “Delete a virtual storage object” task.

Products

• VMware vCenter Server
• Tanzu Kubernetes Runtime

Issue/Introduction

• When vCenter is integrated with Cloud Native Storage (CNS), repeated and failing "Delete a Virtual Storage Object" tasks are seen.
• These tasks fail because the vStorageObject , which represents a persistent volume provisioned by CNS, cannot be deleted.

• /var/log/vmware/vsan-health/vsanvcmgmtd.log point that the datastore path is invalid. 
  
  YYYY-MM-DDTHH:MM:SS.173+01:00 info vsanvcmgmtd[352080] [vSAN@6876 sub=CnsTask] A com.vmware.cns.vslm.tasks.retrieveVStorageObject task is created: task-12345
YYYY-MM-DDTHH:MM:SS.177+01:00 info vsanvcmgmtd[352080] [vSAN@6876 sub=vmomi.soapStub[5]] SOAP request returned HTTP failure; <<io_obj p:0x00007f23f474ad40, h:31, <TCP '127.0.0.1 : 52568'>, <TCP '127.0.0.
1 : 1080'>>, /sdk>, method: retrieveDiskPathAssociations; code: 500(Internal Server Error); fault: (vmodl.fault.SystemError) {
-->    faultCause = (vim.fault.InvalidDatastorePath) {
-->       faultCause = (vmodl.MethodFault) null,
-->       faultMessage = <unset>,
-->       datastore = <unset>,
-->       name = <unset>,
-->       datastorePath = "[Datastore] fcd/###############################.vmdk"
-->       msg = "Invalid datastore path '[Datastore] fcd/###############################.vmdk'."."
-->    },
-->    faultMessage = <unset>,
-->    reason = "Undeclared fault"
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f23f474ad40, h:31, <TCP '127.0.0.1 : 52568'>, <TCP '127.0.0.1 : 1080'>>, /sdk>]: retrieveDiskPathAssociations

Environment

vCenter Server 8.x
VMware cloud native storage (CNS)

Cause

This issue occurs when a datastore containing First Class Disks (FCDs) backing Cloud Native Storage (CNS) Persistent Volumes is renamed in vCenter Server.
When the datastore is renamed, the path recorded in the CNS catalog becomes stale. Subsequent CNS synchronization, health checks, or deletion tasks attempting to query the storage object via the retrieveDiskPathAssociations API will fail because the original path no longer exists.

Resolution

Workaround:

1. Take a snapshot of the vCenter VM.
   
   Enhanced Linked Mode (ELM): If multiple vCenter Server systems are in ELM, perform a simultaneous powered-off snapshot of all linked nodes.
   Standalone vCenter: For standalone deployments, a non-memory snapshot is recommended prior to implementing any changes.
   
   Snapshot Best practices for vCenter Server Virtual Machines
   
   
2. SSH to vCenter as root user
   
   
3. Stop vpxd service
   
   service-control --stop vpxd
   
   
4. Connect to vCenter postgres database
   
   psql -U postgres -d VCDB
   
   
5. Get the required datastore URL from vpx_ds_info or cns.vpx_storage_datastore_info

select name, url from vpx_ds_info;
   
   
6. Check the vclock value of the datastore
   
   select * from cns.vpx_storage_datastore_info;
   
   
7. Reset the datastore vclock to -1 in the database
   
   update cns.VPX_STORAGE_DATASTORE_INFO set vclock=-1 where datastore_url='<ds-url>';
   
   
8. Delete all volume entries for the concerned datastore from the database
   
   delete from cns.volume_info where datastore='<ds-url>';
   
   
9. Exit the VCDB
   
   \q
   
   
10. Start vpxd service
    
    service-control --start vpxd
    
    
11. Restart vsan-health
    
    vmon-cli -r vsan-health
    
    
12. After 1-2 minutes post vsan-health restart, check the database again and the volumes should appear in the database with the correct diskPaths.

select * from cns.volume_info;