Storage vMotion of VM fails with error "Could not complete network copy for file"

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Storage vMotion of a VM with multiple disks attach fails with an error while copying some disks :

Error : Could not complete network copy for file

• Taking backup of such VM using third party software also fails.
• Cloning a VM fails with error: "Error caused by file /vmfs/volumes/########-######/VM_Name/VM_Name-000002.vmdk"
• Manual copy of the vmdk from the vSphere UI to another datastore fails with following error as seen in the recent task view below :

 

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The copy operation failure on some vmdk is due to I/O errors encountered while reading certain blocks of data from the disk.

Reference:

• In the /var/log/vmkernel.log, this error is observed which indicates that ESXi was unable to read a certain  block of data from the virtual disk. This typically points to some corruption within the disk:

        cpu85:2104197)BC: 414: read from VM-flat.vmdk (b00f 53 662b602b d0ba541e 57eedeaf 5fd361af 16f02 0 8e5c8000 1 18ec50000 102000000080000 8200 0) 131072 bytes failed: I/O error

• In the hostd.log file of the ESXi host,this error message indicates that the  ESXi host attempted to clone a VM’s virtual disk, but the operation failed because the underlying disk returned an I/O error. This usually indicates the source VMDK issue:
  
   hostd[2103781] [Originator@6876 sub=DiskLib opID=mdy7xkx4-13490589-auto-815em-h5:71920143-0-01-d340 user=vpxuser] DISKLIB-LIB_CLONE   : DiskLibCloneWholeDisk: Failed to clone whole disk: Input/output error(327689).
 hostd[2103781] [Originator@6876 sub=DiskLib opID=mdy7xkx4-13490589-auto-815em-h5:71920143-0-01-d340 user=vpxuser] DISKLIB-LIB_CLONE   : Failed to clone : Input/output error (327689).
 warning hostd[2103781] [Originator@6876 sub=Libs opID=mdy7xkx4-13490589-auto-815em-h5:71920143-0-01-d340 user=vpxuser] [NFC ERROR]Nfc_DiskLib_Clone: Failed to create VMFSEx2 disk /vmfs/volumes/########-########/Clone/Clone_3.vmdk :   Input/output error
 warning hostd[2103781] [Originator@6876 sub=Libs opID=mdy7xkx4-13490589-auto-815em-h5:71920143-0-01-d340 user=vpxuser] [NFC ERROR]NfcFileDskClone: Failed to clone disk at destination /vmfs/volumes/########-########/Clone/Clone_3.vmdk:   Input/output error (327689)
 hostd[2157529] [Originator@6876 sub=DiskLib opID=m8ofr3o4-1914914-auto-151k3-h5:70716603-e6-01-f865 user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB_CLONE   : DiskLibCopyDataInt failed with Input/output error.
 hostd[2157529] [Originator@6876 sub=DiskLib opID=m8ofr3o4-1914914-auto-151k3-h5:70716603-e6-01-f865 user=vpxuser:VSPHERE.LOCAL\Administrator] DISKLIB-LIB_CLONE   : DiskLibCloneWholeDisk: Failed to clone whole disk: Input/output error(327689).

Resolution

Engage the storage vendor to address recurring I/O errors detected on the disk.