Cross vCenter vMotion or clone task fails

Products

• VMware vSphere ESXi

Issue/Introduction

When you perform a Cross vCenter vMotion or clone a virtual machine with disks larger than 1TB, the migration fails consistently. You observe the following symptoms:

• The migration process stops after approximately 2 hours and 12 minutes.
• The task fails specifically during the transfer of large virtual disks (VMDKs).
• You see the following error in the vCenter Server tasks: Could not complete network copy for file ####/####.vmdk.
• In the source ESXi host /var/run/log/hostd.log, you find entries similar to: 
  YYYY-MM-DDTHH:MM:SS.266Z In(166) Hostd[2098914]: [Originator@6876 sub=NfcManager opID=mhj18a3x-3463-auto-2o8-h5:70000412-58-01-66fd sid=527a7005 user=vpxuser:Domain\user] Initiating transfer for copy spec [ds:///vmfs/volumes/<VM Name>/<VM Name>.vmdk -> ds:///vmfs/volumes//<VM Name>/<VM Name>.vmdk]
.....
  vMotion failed after: 2 hours and 12 minutes with following error
  .....
YYYY-MM-DDTHH:MM:SS.627Z In(166) Hostd[2098914]: [Originator@6876 sub=Libs opID=mhj18a3x-3463-auto-2o8-h5:70000412-58-01-66fd sid=527a7005 user=vpxuser:Domain\user] [NFC ERROR]NfcTransport_FlushDisk: Failed to flush IOs on disk: Out of memory
  
  

Environment

vSphere ESX 8.x

Cause

The vMotion control session times out and closes the connection. When you copy a large disk via Network File Copy (NFC), no data is sent over the control channel while the data transfer is in progress.
If your network path includes firewalls or load balancers with idle connection TCP keepalive timeouts (often set to 2 hours), these devices forcefully terminate the connection due to inactivity on the control channel.

Resolution

This issue will be resolved in upcoming patch.

Workaround
You must reduce the time required for the disk copy or bypass the network devices enforcing the idle timeout:

• Migrate the virtual machine files manually via SCP between hosts, though this may result in thick-provisioned disks on the destination.
• Increase the idle connection timeout settings on physical firewalls or network appliances located between the source and destination management networks.