Troubleshooting vMotion fails with network errors

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

You see errors similar to:

Example 1:

The VMotion failed because the ESX hosts were not able to connect over the VMotion network. Please check your VMotion network settings and physical network configuration. VMotion [184329483:<mig_id>] failed to create connection with remote host <###.###.###.###>: The ESX hosts failed to connect over the VMotion network Migration [184329483:<mig_id>] failed to connect to remote host <###.###.###.###>: Timeout

Example 2:

The VMotion failed because the ESX hosts were not able to connect over the VMotion network. Please check your VMotion network settings and physical network configuration. VMotion [-1408237366:<mig_id>] failed to create connection with remote host <###.###.###.###>: The ESX hosts failed to connect over the VMotion network Migration [-1408237366:<mig_id>] failed to connect to remote host <###.###.###.###>: Connection refused

Example 3:

The vMotion failed because the destination host did not receive data from the source host on the vMotion network. Please check your vMotion network settings and physical network configuration and ensure they are correct.

In the /var/log/hostd.log file, you see entries similar to:

[27703B90 verbose 'Vmsvc.vm:/vmfs/volumes/<datastore_uuid>/TEMP-TEMP/TEMP-TEMP.vmx'] Handling message _vmx1: Migration [a991333:<mig_id>] failed to connect to remote host from host : Timeout

Please note that the "###.###.###.###" text represents IP addresses, which have been anonymized here for security reasons.

Example 4:

The migration was canceled because the amount of changing memory for the virtual machine was greater than the available network bandwidth.

The above error message indicates that the vMotion or DRS operation is unable to complete due to network-related issues, rather than actual memory changes or bandwidth limitations.

Example 5:

Migration [vMotion Migration ID] failed to connect to remote host <Target vMotion VMKernel Interface> from host <Source vMotion VMKernel Interface>: Timeout. vMotion migration [Motion Migration ID] failed to create a connection with remote host <Target vMotion VMKernel Interface>: The ESX hosts failed to connect over the VMotion network The vMotion migrations failed because the ESX hosts were not able to connect over the vMotion network. Check the vMotion network settings and physical network configuration.

Example 6:

Migration [vMotion Migration ID] failed to connect to remote host <Target vMotion VMKernel Interface> from host <Source vMotion VMKernel Interface>: Network Unreachable.

The above suggest an issue with the vSphere vMotion network configuration, such as trying to communicate between a source and destination IP address that are on different subnets, which is not allowed in default configurations.

Example 7:

Migration [184329483:<mig_id>] failed to connect to remote host <###.###.###.###>: Timeout.

This error indicates that remote host did not accept the connection within the allowed time limit.

Example 8:

Migration [-1408237366:<mig_id>] failed to connect to remote host <###.###.###.###>: Connection refused.

This error indicates that the remote host is explicitly not listening on the vMotion port. The vMotion port is 8000, see also Port requirements for VMware vSphere ESXi

Environment

VMware vSphere ESXi

VMware vCenter Server

Cause

The network configuration (either virtual or physical) present is invalid.
Network misconfiguration can cause random vMotion failures. Retrying the vMotion operation may be successful, but it is recommended to follow this article to isolate and correct the problem.

Resolution

To resolve this issue: In addition to the steps below, first, ensure that the required VLANs for vMotion traffic are allowed on the physical switch ports connected to the source and destination ESXi hosts' VMNICs.

Check for IP address conflicts on the vMotion network.

Note: Each host in the cluster should have a vMotion vmkernel, assigned a unique IP address. Additionally, vMotion should only be enabled on one subnet; having vMotion enabled on the vMotion vmkernel adapter and the Management vmkernel adapter both, for instance, can cause issues where the source and destination vMotion hosts are trying to establish a connection on different subnets.
Check for packet loss over the vMotion network. Try having the source host ping (vmkping) the destination host's vMotion vmknic IP address for the duration of the vMotion.

For example:

vmkping -I vmk# <IP of destination host vMotion vmk>

Where vmk# is the vmkernel number of the vMotion vmkernel on the source host.

If jumbo frames has been enabled on the vMotion VMK, test with the Do Not Fragment and payload size switches.

# vmkping -d -s 8972 -I vmk# IP

Where 8972 is the payload size of a ping request with frame size 9000

Where vmk# is the transmit interface vMotion

Where IP is the destination IP of the respective remote vMotion interface

example:

# vmkping -d -s 8972 -I vmk2 10.x.x.xx

Alternatively, where the vMotion vmkernel is configured for 1500 MTU on both source and destination hosts, the below command can be used:
# vmkping -d -s 1472 -I vmk2 10.x.x.xx

If the vMotion IP Stack is configured, use the stack switch.

# vmkping -I vmk# -S <stack-name> <IP>

example:

# vmkping -I vmk2 -S vmotion 10.x.x.xx

For more information, see Testing VMkernel network connectivity with the vmkping command.
Check the driver/firmware of the physical adapter used for vMotion.

Note: If the driver or firmware is not up to date, the vMotions can fail with a "timeout" or "waiting for data" error. For more information, see Determining Network/Storage firmware and driver version in ESXi.
Ensure the ESXi firewall ruleset for vMotion is enabled and that there are no other conflicting rules blocking the traffic.
Check for potential interaction with firewall hardware or software that prevents connectivity between the source and the destination TCP port 8000.

To test connectivity over port 8000:

From source ESXi host, run this command:

# nc -z #.#.#.2 8000

From destination ESXi host, run this command:

# nc -z #.#.#.1 8000

Sample successful output:

# Connection to #.#.#.2 8000 port [tcp/*] succeeded!

Additional Information

For more information on vMotion operations and troubleshooting vMotion issues, see Understanding and troubleshooting vMotion.

Additional Articles

Troubleshooting vMotion network connectivity issues