CNS tasks in vCenter Server frequently fail with the error Sync volume

Products

• VMware vCenter Server
• VMware vSphere ESXi 8.0
• VMware Telco Cloud Platform

Issue/Introduction

After upgrading to vCenter 8.x, you see frequent tasks related to Cloud Native Storage (CNS) fail with the error: Sync Volume error

vpxd.log:

####-##-##T##:##:##.564Z error vpxd[26341] [Originator@6876 sub=Default opID=1d755ca6-2f] [VpxLRO] -- ERROR lro-139956 -- ####-####-####-####-####(uuid) -- VStorageObjectManager -- vim.vslm.vcenter.VStorageObjectManager.retrieveVStorageObjectMetadata: :vmodl.fault.SystemError
--> Result:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Failed to lock the file: api = DiskLib_Open, _diskPath->CValue() = /vmfs/volumes/##/fcd/##/####.vmdk"
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f900c7259a8, h:74, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-####/vpxa>]: retrieveVStorageObjectMetadata

Environment

Vcenter 8.0.1, 8.x

ESXi 8.0.1
TCP 5.0.1

Cause

After upgrading to vCenter 8.x, CNS runs a full sync on datastores to clean up 

Resolution

To resolve the issue:

• If you use both file and block volume in your environment, refer KB 399208
• If there is no record found for volume_id start with the pattern ""file:" then please refer below to resolve:
  • In the VPXD log, you will see entries similar to below:

  • ->    msg = "Received SOAP response fault from [<<io_obj p:0x00007f900c7259a8, h:74, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-####/vpxa>]: retrieveVStorageObjectMetadata

  • Refer the link in the logs and we see a "host-####" which may be having some stale lock file.
  • To address this, make a note of  "host-XXX" ; then go back to Vcenter UI and select each host and refer URL section that matches with the host reported in the logs.
  • Perform the management service restart to remove stale lock file. Restarting Management Agents in ESXi