vMotion Migration Fails with Timeout Due to MTU Size Mismatch

Products

VMware vSphere ESXi

Issue/Introduction

This article addresses a common issue where a VMware vMotion operation fails with a timeout error, directly attributed to a Maximum Transmission Unit (MTU) size mismatch across the network path used for vMotion traffic. This typically occurs when Jumbo Frames are configured inconsistently, leading to packet drops and severe performance degradation during the high-volume data transfer required for vMotion.

vMotion operation fails with a generic network error or specifically mentions a timeout.

Error messages in vCenter or ESXi host logs (e.g., /var/log/vmkernel.log, /var/log/hostd.log) may indicate "Timeout waiting to send data on tcp socket" "connection timed out," "lost connection to peer," or issues with network packets during migration.
vmkping tests with large packet sizes (jumbo frames) fail to reach the destination ESXi host's vMotion VMkernel adapter.
Increased vMotion duration that eventually times out.
Error messages in the vmware.log of the affected VM such as:
VMotionStream timed out while waiting for disk 0's queue count to drop below the maximum limit of 32768 blocks. This could indicate either network or storage problems preventing proper block transfer.

Environment

vCenter Server 8.x
ESXi 7.x
ESXi 8.x

Cause

The vMotion failure is caused by an MTU mismatch in the vMotion network path.
Jumbo Frames (MTU 9000) are configured on the ESXi hosts for vMotion traffic, but MTU 9000 is not supported across the entire vMotion network path. As a result, Jumbo Frame packets are dropped while standard MTU packets (1500) are transmitted successfully.
Due to the packet drops, vMotion data transfer is interrupted and TCP send timeout errors are generated, causing the migration to fail.
The issue is confirmed through vmkping testing, where standard-sized packets are transmitted successfully, while Jumbo Frame packets experience 100% packet loss, indicating that end-to-end Jumbo Frame connectivity is not available.
- Jumbo packet validation:

[root@hostname:~] vmkping -I vmk# 10.###.###.## -d -s 8972 -S vmotion
PING 10.###.###.## (10.###.###.##): 8972 data bytes

--- 10.###.###.## ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

- Standard packet validation:

[root@hostname:~] vmkping -I vmk# 10.###.###.## -d -s 1472 -S vmotion
PING 10.###.###.## (10.###.###.##): 1472 data bytes
1480 bytes from 10.###.###.##: icmp_seq=0 ttl=64 time=0.304 ms
1480 bytes from 10.###.###.##: icmp_seq=1 ttl=64 time=0.269 ms
1480 bytes from 10.###.###.##: icmp_seq=2 ttl=64 time=0.269 ms

--- 10.###.###.## ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.269/0.281/0.304 ms

Resolution

Resolving this issue requires identifying the MTU mismatch point and ensuring consistent MTU settings across the entire vMotion network path.

Identify MTU Configuration Discrepancies CRITICAL: Perform these checks on BOTH the source and destination ESXi hosts and all intermediate network devices.

Check vMotion VMkernel Adapter MTU:
1. vCenter UI: Navigate to Host > Configure > VMkernel adapters. Select the vMotion VMkernel adapter (e.g., vmk1 or vmk-vMotion), click Edit, and check the MTU size.
2. ESXi CLI (SSH to host): bash esxcli network ip interface list | grep -A 5 "Name: <vmotion_vmk_name>" Look for the MTU value (e.g., MTU: 9000).
Check vSwitch MTU (for the vMotion Port Group):
1. vDS (vCenter UI): Navigate to Networking > [ Distributed Switch]. Go to Configure > Properties and check the MTU value for the vDS. This MTU applies to all uplinks and port groups on the vDS.
2. vSS (ESXi CLI): bash esxcli network vswitch standard list | grep -A 5 "Name: <vSwitch_name>" Look for the MTU value of the standard switch carrying vMotion traffic.
Check Physical Network Switch Ports:
1. CLI of physical switch: Log in to the physical switch(es) connected to the ESXi host uplinks used for vMotion.
2. Check the configuration of the relevant physical interfaces (access ports, trunk ports, port-channels).
3. Look for MTU settings (mtu 9000 or equivalent).
4. Check if a global Jumbo Frame setting is enabled and configured correctly.
Check Routers (if L3 vMotion):
1. If vMotion is occurring across subnets (Layer 3), ensure all routers in the path have their interfaces configured to support the desired MTU (e.g., 9000).

Perform MTU Path Discovery (vmkping) This step helps pinpoint the exact MTU size that the path can support.

SSH to one of the ESXi hosts (e.g., source host).
Run vmkping with the "Don't Fragment" bit and varying packet sizes.
1. Start with a large packet size (e.g., 8972 bytes) and the vMotion VMkernel adapter: bash vmkping -d -s 8972 -I <vmotion_vmk_name> <destination_vmotion_ip> (Replace <vmotion_vmk_name> with your vMotion VMkernel adapter name, e.g., vmk1, and <destination_vmotion_ip> with the IP of the vMotion VMkernel adapter on the target host).
2. If this fails: Gradually reduce the packet size (e.g., to 8000, 7000, 1500, then to 1472 for standard 1500 MTU without headers) until the vmkping succeeds.
3. The largest packet size that succeeds + 28 bytes (for IP/ICMP headers) is the effective MTU of the path.
4. Example for standard MTU: vmkping -d -s 1472 -I <vmotion_vmk_name> <destination_vmotion_ip> (if this works, the path's effective MTU is 1500).
Repeat from the destination ESXi host to the source ESXi host. This confirms bi-directional path MTU.

Ensure Consistent MTU WARNING: Modifying MTU settings can disrupt network connectivity if not applied consistently across the entire path. Perform during a maintenance window or with caution.

Decide on a Consistent MTU:

a. The simplest and often recommended approach is to enable Jumbo Frames (MTU 9000) end-to-end: This requires all components in the vMotion path to support and be configured for MTU 9000.
b. Revert to Standard MTU (1500) end-to-end: If Jumbo Frames are not strictly necessary or problematic to implement consistently.

2. Apply Consistent MTU Settings (on all components in the path)

VMkernel Adapter MTU: vCenter UI: Edit the vMotion VMkernel adapter and set the MTU to the desired consistent value (e.g., 9000 or 1500).
ESXi CLI: bash esxcli network ip interface set -i <vmotion_vmk_name> -m <desired_mtu> vSwitch MTU:
vDS (vCenter UI): Navigate to Networking > [Your Distributed Switch]. Go to Configure > Properties and set the MTU to the desired value.
vSS (ESXi CLI): bash esxcli network vswitch standard set -v <vSwitch_name> -m <desired_mtu>
Physical Network Switch Ports/Global MTU: Configure all relevant switch interfaces and global MTU settings to match the desired MTU.
Routers (if L3): Configure all router interfaces in the vMotion path to the desired MTU.
Save/Apply Changes: Ensure all changes are saved and applied on all devices. A reboot of ESXi hosts is usually not required for MTU changes, but physical switch changes might require saving and sometimes a module restart depending on the vendor.

Re-run vmkping tests : Confirm that vmkping -d -s <desired_mtu - 28> -I <vmotion_vmk_name> <destination_vmotion_ip> now succeeds in both directions
1. Attempt a Test vMotion
2. Initiate a vMotion of a non-critical VM between the affected ESXi hosts.
3. Monitor the vMotion task to ensure it completes successfully and within expected timeframes.
Note : Dedicated vMotion Network: Always use a dedicated VMkernel port, VLAN, and IP subnet for vMotion traffic to isolate it and simplify troubleshooting.
High Bandwidth: Provide sufficient network bandwidth (10Gbps or higher) for vMotion.
NTP Synchronization: Ensure all ESXi hosts and vCenter Server are synchronized with an accurate Network Time Protocol (NTP) source. Time discrepancies can lead to vMotion failures.
Redundancy: Implement NIC teaming for vMotion-enabled VMkernel ports for redundancy and increased bandwidth.

Additional Information

vMotion Network Requirements

Troubleshooting vMotion issues

Troubleshooting vMotion fails with network errors