Cross vCenter migration(XVMotion)fails with error: Could not complete network copy for file /vmfs/volumes/datastore_uuid/VM_Folder_VM_name.vmdk

Products

• VMware vCenter Server

Issue/Introduction

• Cross vCenter Migration(XVmotion) fails with NFC errors:

Could not complete network copy for file /vmfs/volumes/DataStore_UUID/VM_Folder_VM_name.vmdk 

• We can also observe timeout errors in logs:"timed out while waiting for disk 1's queue count to drop below the maximum limit of 32768 blocks. This could indicate either network or stora"
• /var/log/vmware/vmkwarning.log reports below errors

YYYY-MM-DDTHH:MM:SS.845Z Wa(180) vmkwarning: cpu72:4635412)WARNING: XVMotion: 3311: ########-5c22-10fa-####-############S: timed out while waiting for disk 1's queue count to drop below the maximum limit of 32768 blocks. This could indicate either network or stora$
YYYY-MM-DDTHH:MM:SS.845Z Wa(180) vmkwarning: cpu72:4635412)WARNING: Nfc: 100: ########-5c22-10fa-####-############ S: Failed to allocate the XvMotion blocks to back scatter gather array: Timeout
YYYY-MM-DDTHH:MM:SS.845Z In(182) vmkernel: cpu72:4635412)Migrate: 101: ########-5c22-10fa-####-############ S: MigrateState: Failed
YYYY-MM-DDTHH:MM:SS.845Z Wa(180) vmkwarning: cpu72:4635412)WARNING: Migrate: 257: ########-5c22-10fa-####-############ S: Failed: Out of memory (0xbad0014) @0x420005d13ec8

• /var/log/vmware/vpxd/vpxd.log reports below errors

YYYY-MM-DDTHH:MM:SS.277+05:30 error vpxd[2024583] [Originator@6876 sub=Datastore opID=mk0se8i1-263961-auto-5npx-h5:70089938-34-01] [VpxdDatastore::UrlToDSPath] Received a non-url [/vmfs/volumes/DataStore_UUID
/VM_Name/VM_Name.vmdk], instead of a url
YYYY-MM-DDTHH:MM:SS..277+05:30 error vpxd[2024583] [Originator@6876 sub=VmProv opID=mk0se8i1-263961-auto-5npx-h5:70089938-34-01] Aborting task tracking since task vim.Task:haTask--nfc.NfcManager.copy-8548084 failed
YYYY-MM-DDTHH:MM:SS..277+05:30 error vpxd[2024583] [Originator@6876 sub=VmProv opID=mk0se8i1-263961-auto-5npx-h5:70089938-34-01] Get exception while executing action vpx.vmprov.CopyVmFiles:
--> (vim.fault.NetworkCopyFault) {
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "",
-->          arg = (vmodl.KeyAnyValue) [
-->             (vmodl.KeyAnyValue) {
-->                key = "DestFile",
-->                value = "/vmfs/volumes/DataStore_UUID/VM_Name/VM_Name.vmdk"
-->             },
-->             (vmodl.KeyAnyValue) {
-->                key = "NfcErrorCode",
-->                value = 5
-->             },
-->             (vmodl.KeyAnyValue) {
-->                key = "NfcErrorCodeString",
-->                value = "NFC_NO_MEMORY"
-->             },
-->             (vmodl.KeyAnyValue) {
-->                key = "NfcErrorString",
-->                value = "No more memory is available"
-->             }
-->          ],
-->          message = ""
-->       }
-->    ],
-->    file = "/vmfs/volumes/DataStore_UUID/VM_Name/VM_Name.vmdk",
-->    msg = "Could not complete network copy for file /vmfs/volumes/DataStore_UUID/VM_Name/VM_Name.vmdk"

Environment

Esx 9.x

vCenter Server 9.x

Cause

This issue occurs when there is an MTU mismatch along the data path between the source and destination ESXi hosts.

Resolution

• To resolve this issue ensure the MTU is set to 1500 or 9000 as per the configuration set on physical switch
• Verify the vSwitch Security Policy is same on both Source and Destination Esx hosts. 
• To verify if the MTU is set correctly we can execute command on Source Esxi host: vmkping -I vmk# dest_vmk_IP -d -s 8972   ---> where # is the source vMotion vmkernel adapter.

 

NOTE: If ICMP ping is disabled within the infrastructure, configure the MTU to 1500 on both the source and destination Esx Hosts.

Additional Information

This issue can also be causes by limited (or throttled) bandwidth between locations.  Please see: XvMotion runs very slow and eventually fails

For Additional details refer below documents: Cross vCenter Migration and Clone requirements in VMware vSphere 6.x and later