Relocation or mounting of persistent volumes fails after datastore name change

Products

• VMware vCenter Server

Issue/Introduction

• After a change of datastore name in vSphere 8.0, issue can occur with PVs/FCDs unable to attach/detach.
• Volumes can also be stuck in released state.
• Constant "update container volume" tasks or "Retrieve a virtual storage object" may be shown on the vCenter UI.
• In vCenter vpxd logs, errors related to the created volumes are observed stating retrieveDiskPathAssociations: :vim.fault.InvalidDatastorePath

/var/log/vmware/vpxd.log

error vpxd[2824780] [Originator@6876 sub=Default opID=0788a205-##] [VpxLRO] -- ERROR lro-262708904 -- 5202bc7b-ef27-562e-4a0c-###(5281ac6b-3989-9abc-1c94-###) -- VStorageObjectManager -- vim.vslm.vcenter.VStorageObjectManager.retrieveDiskPathAssociations: :vim.fault.InvalidDatastorePath
--> Result:
--> (vim.fault.InvalidDatastorePath) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    datastore = <unset>,
-->    name = <unset>,
-->    datastorePath = "[<datstore-name>] <path>/<to>/<vmdk>.vmdk"
-->    msg = ""
--> }
--> Args:
-->
--> Arg diskPaths:
--> (string) [
-->    "[datastore-name] <path>/<to>/<vmdk>.vmdk"
--> ]
--> Arg datastore:
--> 'vim.Datastore:datastore-####'
(END)

• In vCenter vsanvcmgmtd logs, following errors are observed

/var/log/vmware/vsan-health/vsanvcmgmtd.log

-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f74e001b1e0, h:4, <TCP '127.0.0.1 : 32972'>, <TCP '127.0.0.1 : 1080'>>, /sdk>]: retrieveDiskPathAssociations
--> Method vim.vslm.vcenter.VStorageObjectManager.retrieveDiskPathAssociations threw undeclared fault of type vim.fault.InvalidDatastorePath"
--> }


Environment

VMware vCenter Server 8.x+

VMware vSphere CSI Driver 3.0+

Cause

If a datastore name is changed from the vCenter, CNS database entries are not updated to reflect the change.

Resolution

Workaround:

1. Take a snapshot of the vCenter VM.
   
   Enhanced Linked Mode (ELM): If multiple vCenter Server systems are in ELM, perform a simultaneous powered-off snapshot of all linked nodes.
   Standalone vCenter: For standalone deployments, a non-memory snapshot is recommended prior to implementing any changes.
   
   Snapshot Best practices for vCenter Server Virtual Machines
   
   
2. SSH to vCenter as root user
   
   
3. Stop vpxd service
   
   service-control --stop vpxd
   
   
4. Connect to vCenter postgres database
   
   psql -U postgres -d VCDB
   
   
5. Get the required datastore URL from vpx_ds_info or cns.vpx_storage_datastore_info

select name, url from vpx_ds_info;
   
   
6. Check the vclock value of the datastore
   
   select * from cns.vpx_storage_datastore_info;
   
   
7. Reset the datastore vclock to -1 in the database
   
   update cns.VPX_STORAGE_DATASTORE_INFO set vclock=-1 where datastore_url='<ds-url>';
   
   
8. Delete all volume entries for the concerned datastore from the database
   
   delete from cns.volume_info where datastore='<ds-url>';
   
   
9. Exit the VCDB
   
   \q
   
   
10. Start vpxd service
    
    service-control --start vpxd
    
    
11. Restart vsan-health
    
    vmon-cli -r vsan-health
    
    
12. After 1-2 minutes post vsan-health restart, check the database again and the volumes should appear in the database with the correct diskPaths.

select * from cns.volume_info;


Additional Information

Issue will be resolved in an upcoming release of vSphere.