In vCenter Server, vslm.vcenter.vStorageObjectManager.updatevStorageObjectMetadataEx.label tasks are failing

Products

• VMware vSphere Kubernetes Service
• VMware vCenter Server

Issue/Introduction

In vCenter Server, the vslm.vcenter.vStorageObjectManager.updatevStorageObjectMetadataEx.label tasks are failing. The VPXD logs will contain the following entries related to the affected datastore.

vpxd[108017] [Originator@6876 sub=MoVStorageObjectMgr opID=6f5cdd91-1f] For FCD ########, check if user has been granted permission on datastore vim.Datastore:datastore-###### 

vpxd[06117] [Originator@6876 sub=vpxLro opID=6ee0bd36-6c] [VpxLRO] -- BEGIN task-5063008 -- VStorageObjectManager -- vim.vslm.vcenter.VStorageObjectManager.updateVStorageObjectMetadataEx -- 52675237-5c3a-8983-0066-82e4fb90c8e6(5260e571-9fd7-9d0d-ca84-8dd8c7ae0db7)

vpxd[06117] [Originator@6876 sub=MoVStorageObjectMgr opID=6ee0bd36-6c] Updating VStorageObject: id = ########, datastore = vim.Datastore:datastore-######, returnFcd = true 

vpxd[06117] [Originator@6876 sub=vpxLro opID=6ee0bd36-6c] [VpxLRO] -- FINISH task-5063008

error vpxd[06117] [Originator@6876 sub=Default opID=6ee0bd36-6c] [VpxLRO] -- ERROR task-5063008 -- 52675237-5c3a-8983-0066-82e4fb90c8e6(5260e571-9fd7-9d0d-ca84-8dd8c7ae0db7) -- VStorageObjectManager -- vim.vslm.vcenter.VStorageObjectManager.updateVStorageObjectMetadataEx: :vmodl.fault.InvalidArgument

Environment

vSphere with Tanzu 
VMware vCenter Server

Cause

The FCD catalog reconciliation is failing on the datastore.

Resolution

The datastore catalog requires reconciliation. Please refer to the KB article for instructions: 

• Reconciling Discrepancies in the Managed Virtual Disk Catalog (2147750)
• Additionally, the CNS service needs to be restarted. Use the following command:
  

  service-control --stop vmware-vsan-health && service-control --start vmware-vsan-health

Additional Information

If the task failed with result "The object or item referred to could not be found” as following, please refer VCDB is out of sync with datastore catalog

"VStorageObjectManager -- vim.vslm.vcenter.VStorageObjectManager.updateVStorageObjectMetadataEx: :vim.fault.NotFound
--> Result:
--> (vim.fault.NotFound) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f38100709c8, h:109, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-xxxx/vpxa>]: retrieveVStorageObjectPathAndCrypto
--> Received SOAP response fault from [<<io_obj p:0x0000006606aabb80, h:23, <TCP '127.0.0.1 : 40792'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: retrieveVStorageObjectPathAndCrypto
--> The object or item referred to could not be found."
--> }
--> Args:
-->
--> Arg id:
--> (vim.vslm.ID) {
-->    id = "<CNS volumd ID>"