Troubleshooting vMotion fails with network errors
search cancel

Troubleshooting vMotion fails with network errors

book

Article ID: 318636

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vCenter Server

Issue/Introduction

You see errors similar to:

Example 1:

The VMotion failed because the ESX hosts were not able to connect over the VMotion network. Please check your VMotion network settings and physical network configuration. VMotion [184329483:<mig_id>] failed to create connection with remote host <###.###.###.###>: The ESX hosts failed to connect over the VMotion network Migration [184329483:<mig_id>] failed to connect to remote host <###.###.###.###>: Timeout

Example 2:

The VMotion failed because the ESX hosts were not able to connect over the VMotion network. Please check your VMotion network settings and physical network configuration. VMotion [-1408237366:<mig_id>] failed to create connection with remote host <###.###.###.###>: The ESX hosts failed to connect over the VMotion network Migration [-1408237366:<mig_id>] failed to connect to remote host <###.###.###.###>: Connection refused

Example 3:

The vMotion failed because the destination host did not receive data from the source host on the vMotion network. Please check your vMotion network settings and physical network configuration and ensure they are correct.

In the /var/log/hostd.log file, you see entries similar to:

[27703B90 verbose 'Vmsvc.vm:/vmfs/volumes/<datastore_uuid>/TEMP-TEMP/TEMP-TEMP.vmx'] Handling message _vmx1: Migration [a991333:<mig_id>] failed to connect to remote host from host : Timeout

Please note that the "###.###.###.###" text represents IP addresses, which have been anonymized here for security reasons.

Example 4:

The migration was canceled because the amount of changing memory for the virtual machine was greater than the available network bandwidth.
The above error message indicates that the vMotion or DRS operation is unable to complete due to network-related issues, rather than actual memory changes or bandwidth limitations.
 
Example 5:
 
Migration [vMotion Migration ID] failed to connect to remote host <Target vMotion VMKernel Interface> from host <Source vMotion VMKernel Interface>: Timeout. vMotion migration [Motion Migration ID] failed to create a connection with remote host <Target vMotion VMKernel Interface>: The ESX hosts failed to connect over the VMotion network The vMotion migrations failed because the ESX hosts were not able to connect over the vMotion network. Check the vMotion network settings and physical network configuration.

Environment

VMware vSphere ESXi

VMware vCenter Server

Cause

The network configuration (either virtual or physical) present is invalid.
Network misconfiguration can cause random vMotion failures. Retrying the vMotion operation may be successful, but it is recommended to follow this article to isolate and correct the problem.

Resolution

To resolve this issue:  In addition to the steps below, first, ensure that the required VLANs for vMotion traffic are allowed on the physical switch ports connected to the source and destination ESXi hosts' VMNICs.

  1. Check for IP address conflicts on the vMotion network.

    Note: Each host in the cluster should have a vMotion vmkernel, assigned a unique IP address.
     
  2. Check for packet loss over the vMotion network. Try having the source host ping (vmkping) the destination host's vMotion vmknic IP address for the duration of the vMotion.

    For example:

    vmkping -I vmk# <IP of destination host vMotion vmk>

    Where vmk# is the vmkernel number of the vMotion vmkernel on the source host.

    If jumbo frames has been enabled on the vMotion VMK, test with the Do Not Fragment and payload size switches.
    # vmkping -d -s 8972 -I vmk# IP
    Where 8972 is the payload size of a ping request with frame size 9000
    Where vmk# is the transmit interface vMotion
    Where IP is the destination IP of the respective remote vMotion interface
    example:
    # vmkping -d -s 8972 -I vmk2 10.x.x.xx
     

    Alternatively, where the vMotion vmkernel is configured for 1500 MTU on both source and destination hosts, the below command can be used: 
    # vmkping -d -s 1472 -I vmk2 10.x.x.xx
    If the vMotion IP Stack is configured, use the stack switch.
    # vmkping -I vmk# -S <stack-name> <IP>
    example:
    # vmkping -I vmk2 -S vmotion 10.x.x.xx

    For more information, see Testing VMkernel network connectivity with the vmkping command.
     
  3. Check the driver/firmware of the physical adapter used for vMotion.

    Note: If the driver or firmware is not up to date, the vMotions can fail with a "timeout" or "waiting for data" error. For more information, see Determining Network/Storage firmware and driver version in ESXi.
     
  4. Ensure the ESXi firewall ruleset for vMotion is enabled and that there are no other conflicting rules blocking the traffic. 
  5. Check for potential interaction with firewall hardware or software that prevents connectivity between the source and the destination TCP port 8000.

Additional Information

For more information on vMotions and troubleshooting vMotion issues, see Understanding and troubleshooting vMotion.

Other examples:

  • Migration [184329483:<mig_id>] failed to connect to remote host <###.###.###.###>: Timeout.

    This error indicates that remote host did not accept the connection within the allowed time limit.
     
  • Migration [-1408237366:<mig_id>] failed to connect to remote host <###.###.###.###>: Connection refused.

    This error indicates that the remote host is explicitly not listening on the vMotion port.  The vMotion port is 8000, see also Port requirements for VMware vSphere ESXi

Additional Articles

Powered by Wolken Software