Cross vCenter vMotion Fails with Timeout Error Due to MTU Mismatch

Products

• VMware vSphere ESXi

Issue/Introduction

• Cross vCenter vMotion migrations fail during the memory transfer phase.

• The migration progress reaches a certain percentage and then stalls, eventually failing with a timeout error.

• Error Message in vCenter UI: Migration to host <Destination_IP> failed with error Timeout (195887137).



• Error Message in the source ESXi host logs (/var/run/log/vmkernel.log): "Migration to host <Destination-host_vMotion_vmkernel_IP_address> failed with error Timeout (195887137). <Timestamp> vMotion migration [182390150:6593197149952505377] (16-73770624867792) failed to receive 4980/4980 bytes from the remote host <Source-host_vMotion_vmkernel_IP_address>: Timeout Failed waiting for data. Error 195887137. Timeout. <Timestamp> vMotion migration [182390150:6593197149952505377] failed to send buffer to remote host: Timeout. vMotion migration [182390150:6593197149952505377] failed writing stream completion: Timeout".
  
  

• Errors in vmkernel.log on Destination host:
  • • failed to receive ####/#### bytes from the remote host : Timeout
    • NfcNewAuthdConnectionEx: Failed to connect
    • failed to read stream keepalive: Connection reset by peer

Environment

• VMware vSphere ESXi 7.x, 8.x, 9.x
• VMware vCenter Server 7.x, 8.x, 9.x

Cause

The issue is caused by an inconsistent Path Maximum Transmission Unit (PMTU) across the physical and virtual network infrastructure.

While the ESXi hosts are configured for Jumbo Frames (MTU 9000) on the vMotion VMkernel interface (vmk) and the Virtual Distributed Switch (VDS), the physical network path (switches, routers, or firewalls) is not configured to support the 9000-byte payload. This results in the silent dropping of large packets that exceed the physical network's MTU limit, leading to a synchronization timeout between the source and destination hosts.

Resolution

To resolve this issue, ensure the MTU configuration is consistent across the entire end-to-end network path. Use one of the following two methods:

Method 1: Enable Jumbo Frames End-to-End (Recommended)

1. Physical Infrastructure: Consult with your Network Administration team to enable Jumbo Frames (MTU 9000 or 9216) on all physical switch ports, trunk lines, and any firewalls situated between the source and destination vMotion subnets.

2. vSphere Configuration: Ensure the MTU is set to 9000 on:

   • The Virtual Distributed Switch (VDS).

   • The vMotion VMkernel adapter (vmk) on all participating hosts.

3. Validation: Run a ping test with the "Do Not Fragment" bit set to verify the path: vmkping -I vmkX -d -s 8972 <Destination_IP> (Replace vmkX with your vMotion VMkernel ID)

Method 2: Revert to Standard MTU (Workaround)

If the physical network cannot be configured for Jumbo Frames:

1. Change the MTU setting to 1500 on the vMotion VMkernel adapters of all affected ESXi hosts.

2. Change the MTU setting to 1500 on the Virtual Distributed Switch (VDS).

3. Validation: Run a standard ping test to verify connectivity: vmkping -I vmkX <Destination_IP>

Additional Information

For validation of cross-vCenter vMotion network requirements, please refer to: https://knowledge.broadcom.com/external/article/393932