After a reboot, vMotion fails with the error "The vMotion failed because the destination host did not receive data from the source host on the vMotion network."

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• vMotion fails with the following error message:

The vMotion failed because the destination host did not receive data from the source host on the vMotion network. Please check the vMotion network setting and physical network configuration and ensure they are correct. "[########################]" Network unreachable. vMotion migration.
The ESX hosts failed to connect over the VMotion network. The vMotion migrations failed because the ESX hosts were not able to connect over the vMotion network. Check the vMotion network settings and physical network configuration.
[YYYY-MM-DDTHH:MM] vMotion migration [########################]  failed to read stream keepalive: Connection closed by remote host, possibly due to timeout.

• After an ESXi host is upgraded or rebooted, vMotion may fail with the error above.

Environment

VMware ESXi

VMware vCenter Server

Cause

vMotion failures can occur due to various inconsistencies or connectivity issues in the virtual or physical network between the source and destination ESXi hosts.

Resolution

 To troubleshoot the problem, it's essential to validate the network connectivity between the hosts:

1. Check the IP connectivity over vMotion kernel interface between source and destination ESXi hosts to make sure if they are reachable with expected MTU: 
   •  vmkping -I vmkX #.#.#.# -d -s XXXX

-I is to specify the VMkernel interface used for vMotion on the source ESXi host.
vmkX is the vmkernel adapter for vMotion on the source ESXi host.
#.#.#.# is the destination host's vMotion vmkernel adapter IP address.
-d ensures the packets are not fragmented during the vmkping operation.
-s is to specify the packet size of the ping packets; ensure the size used matches the MTU size configured on the source and destination hosts' vmkernel adapters (1472 for 1500 MTU, 8972 for 9000 MTU).
2. Based on point 1 if the vmkpings are failing, perform packet captures while initiating vmkpings as mentioned in step 1 and check the drop point of packets in the network.
   Refer to the following KB article for performing packet captures: Packet capture on ESXi using the pktcap-uw tool
3. Based on the packet captures performed in step 2, if the packets are dropping in the physical network, to resolve the issue, engage the physical network team/administrator.
4. For example, to test with 1500 MTU, run the command:
   • vmkping -I vmkx #.#.#.# -d -s 1472
5. Alternatively, run the command below to test 9000 MTU:
   • vmkping -I vmkx #.#.#.# -d -s 8972
6. If the vMotion VMkernel adapter is configured on the vMotion TCP/IP stack, run the following command to clarify the correct netstack:
   • vmkping -I vmkX #.#.#.# -d -s 1472 -S vmotion 
7. Check for potential issues with connectivity of vmotion network over port TCP 8000.
   
   • nc -zv <IP of destination host vMotion vmkernel adapter> 8000
8. If there is a VLAN in place and used for vMotion, ensure that this VLAN is inbound on the NIC(s)/vmnics that are used for vMotion. 

 

Temporary workarounds:

• Validate if vmkping is successful for the Management network for the source and destination. 

If yes, configure vMotion to operate over the Management network (vmk0) 

vMotion Network --> Edit --> Uncheck vMotion box 

Management Network --> Edit --> Check vMotion box

• In a multi-NIC vMotion configuration, remove either the active uplink or the standby uplink to further isolate the issue.

 

Additional Information

For more information on vMotion and troubleshooting vMotion issues, see Understanding and troubleshooting vMotion (321009)