Relocation or mounting of persistent volumes fail after datastore name change

Products

• VMware vCenter Server

Issue/Introduction

After a change of datastore name in vSphere 8.0 U2, issue can occur with PVs/FCDs unable to attach/detach. Volumes can also be stuck in released state.

Constant update container volume tasks may be shown on the vCenter UI.

In vpxd logs, we would see errors relating to created volumes stating retrieveDiskPathAssociations: :vim.fault.InvalidDatastorePath

/var/log/vmware/vpxd.log

error vpxd[2824780] [Originator@6876 sub=Default opID=0788a205-86] [VpxLRO] -- ERROR lro-262708904 -- 5202bc7b-ef27-562e-4a0c-***(5281ac6b-3989-9abc-1c94-***) -- VStorageObjectManager -- vim.vslm.vcenter.VStorageObjectManager.retrieveDiskPathAssociations: :vim.fault.InvalidDatastorePath
--> Result:
--> (vim.fault.InvalidDatastorePath) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    datastore = <unset>,
-->    name = <unset>,
-->    datastorePath = "[<datstore-name>] <path>/<to>/<vmdk>.vmdk"
-->    msg = ""
--> }
--> Args:
-->
--> Arg diskPaths:
--> (string) [
-->    "[datastore-name] <path>/<to>/<vmdk>.vmdk"
--> ]
--> Arg datastore:
--> 'vim.Datastore:datastore-1082'
(END)

Environment

Version 8.0 U2

Cause

If a datastore name is changed, CNS database entries is not updated to reflected the change.

Resolution

Please ensure valid snapshot and/or backup has been taken of VCSA before completing steps.

Workaround steps are:
1. Login to VC via SSH.

2. Stop vpxd

service-control --stop vpxd

3. Connect to postgres

psql -U postgres -d VCDB

3. Delete all volumes for the concerned datastore from DB.

delete from cns.volume_info where datastore='<ds-url>';

4. Reset the datastore vclock to -1 in the DB.

update cns.VPX_STORAGE_DATASTORE_INFO set vclock=-1 where datastore_url='<ds-url>';

5. Start vpxd

    service-control --start vpxd

6. Restart vsan-health

vmon-cli -r vsan-health

After 1-2 minutes post restart, check the DB again and the volumes should appear in the DB with the correct diskPaths.

select * from cns.volume_info;

Additional Information

Engineering are aware of this issue and are working towards a fix in a future release.