Upload Files to vSAN Datastore not possible

Products

• VMware vCenter Server

Issue/Introduction

• You are unable to upload files in vSAN datastore with error message 

   Failed to transfer data

• Clones and deployments from citrix on this datastore also fail
• Upload from ESXI host client is working
• In vCenter /var/log/vmware/vpxd.log file you can see lines similar to

<timestamp> error vpxd[3154035] [Originator@6876 sub=Default opID=xxxxx] [VpxLRO] -- ERROR lro-104956 -- xxxxxxx -- nfcService -- vim.NfcService.fileManagement: :vim.fault.NotFound
--> Result:
--> (vim.fault.NotFound) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007fd75812a228, h:73, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-##/vpxa>]: retrieveVStorageObject
--> Received SOAP response fault from [<<io_obj p:0x00000036988bb6b0, h:19, <TCP '127.0.0.1 : ####'>, <TCP '127.0.0.1 : 8307'>>, /sdk>]: retrieveVStorageObject
--> The object or item referred to could not be found."
--> }
--> Args:
-->
--> Arg ds:
--> 'vim.Datastore:<id>:datastore-####'
--> Arg hostForAccess:
--> 'vim.HostSystem:<id>:host-xx'
--> Arg files:
--> (string) [
-->    "<path>/<virtualdiskname>.vmdk"
--> ]
--> Arg mask:
--> 0

 

• In esxi  corresponding to  /hgw/host-##/ log file /var/log/hostd.log , you can see lines similar to:

<timestamp> In(166) Hostd[526093]: [Originator@6876 sub=DiskLib opID=<opid> sid=<sid> user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB   : Failed to open '/vmfs/volumes/<vsan_datstore_uuid>/<directory>/<virtual_disk_name>.vmdk' with flags 0x110001 The system cannot find the file specified (25).

<timestamp> Wa(164) Hostd[526093]: [Originator@6876 sub=Default opID=<opid> sid=<sid> user=vpxuser:VSPHERE.LOCAL\Administrator] Vslm Failure: VslmRetrieveDisk failed for fcd <fcd_name>
on datastore /vmfs/volumes/<vsan_datstore_uuid>/ with type vim.fault.NotFound, N3Vim5Fault8NotFound9ExceptionE(Fault cause: vim.fault.NotFound

<timestamp> In(166) Hostd[526093]: [Originator@6876 sub=Solo.Vmomi opID=<opid> sid=<sid> user=vpxuser:VSPHERE.LOCAL\Administrator] Activation finished; <<#####0, <TCP '127.0.0.1 : 8
307'>, <TCP '127.0.0.1 : ####'>>, ha-vstorage-object-manager, vim.vslm.host.VStorageObjectManager.retrieveVStorageObject, <vim.version.v8_0_3_0, internal, 8.0.3.0>, [N11HostdCommon18VmomiAdapterServer19ActivationResponderE:0x0000004827d
48d88]>
<timestamp> Db(167) Hostd[526093]: [Originator@6876 sub=Solo.Vmomi opID=<opid> sid=<sid> user=vpxuser:VSPHERE.LOCAL\Administrator] Arg id:
<timestamp> Db(167) Hostd[526076]: --> (vim.vslm.ID) {
<timestamp> Db(167) Hostd[526076]: -->    id = "<fcd_name>"
<timestamp> Db(167) Hostd[526076]: --> }
<timestamp> Db(167) Hostd[526093]: [Originator@6876 sub=Solo.Vmomi opID=<opid> sid=<sid> user=vpxuser:VSPHERE.LOCAL\Administrator] Arg datastore:
<timestamp> Db(167) Hostd[526076]: --> 'vim.Datastore:vsan:<vsan_datstore_uuid>
<timestamp> Db(167) Hostd[526093]: [Originator@6876 sub=Solo.Vmomi opID=<opid> sid=<sid> user=vpxuser:VSPHERE.LOCAL\Administrator] Arg diskInfoFlags:
<timestamp> Db(167) Hostd[526076]: --> (string) []
<timestamp> In(166) Hostd[526093]: [Originator@6876 sub=Solo.Vmomi opID=<opid> sid=<sid> user=vpxuser:VSPHERE.LOCAL\Administrator] Throw vim.fault.NotFound
<timestamp> In(166) Hostd[526093]: [Originator@6876 sub=Solo.Vmomi opID=<opid> sid=<sid> sid=5235bc7e user=vpxuser:VSPHERE.LOCAL\Administrator] Result:
<timestamp> In(166) Hostd[526076]: --> (vim.fault.NotFound) {
<timestamp> In(166) Hostd[526076]: -->    msg = "",
<timestamp> In(166) Hostd[526076]: --> }

Environment

vSAN 8.x

vSphere 8.x

Cause

This is due to discrepancies in the Managed Virtual Disk Catalog

Resolution

To resolve the issue, reconcile discrepancies in the Managed Virtual Disk Catalog by using the "for vCenter Server" section ReconcileDatastoreInventory_Task or ScheduleReconcileDatastoreInventory from the following KB

Reconciling Discrepancies in the Managed Virtual Disk Catalog