Cross vCenter vMotion fails with error "The ESX hosts failed to connect over the vMotion network ."

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

Cross vCenter vMotion of VM fails with an error stack similar to: 

• The vMotion failed because the destination host did not receive data from the source host on the vMotion network
  

  Relocate virtual machine The vMotion failed because the destination host did not receive data from the source host on the vMotion network. Please check your vMotion network settings and physical network configuration and ensure they are correct.
  
  Migration [] failed to connect to remote host <#.#.#.#> from host <#.#.#.#>: Timeout. vMotion migration [] failed to create a connection with remote host <#.#.#.#>: The ESX hosts failed to connect over the vMotion network 
  
  The vMotion migrations failed because the ESX hosts were not able to connect over the vMotion network. Check the vMotion network settings and physical network configuration.

• The vMotion failed because the destination host did not receive data from the source host on the vMotion network
  

  The vMotion failed because the destination host did not receive data from the source host on the vMotion network. Please check your vMotion network settings and physical network configuration and ensure they are correct. 
  
  Migration [#] failed to connect to remote host <##.###.#.###> from host <##.###.##.##>: Network unreachable. 
  
  vMotion migration [#] failed to create a connection with remote host <##.###.#.###>: The ESX hosts failed to connect over the VMotion network.
  
  The vMotion migrations failed because the ESX hosts were not able to connect over the vMotion network. Check the vMotion network settings and physical network configuration. 

Ping tests between the vmotion vmkernel interfaces fail:

vmkping -I vmkX #.#.#.# -S vmotion  or   vmkping -I vmkX #.#.#.#  fails with "sendto() failed (Network is unreachable)" or "100% packet loss"

Environment

vCenter Server

vSphere ESXi

Cause

The default gateway setting in TCP/IP configuration or VMkernel for vmotion service differs on both the source and destination hosts.

In cross vCenter migrations involving different subnets, vMotion requires Layer 3 routing. If the vMotion vmkernel interface lacks a default gateway or the physical network blocks port 8000/902, the hosts cannot establish a connection and vMotion will fail.

Resolution

The Default Gateway setting in TCP/IP Configuration or VMkernel needs to be added or adjusted. 

If the vMotion VMkernel Adapter is using the vMotion TCP/IP stack: 

1. Select the host in question
2. Select Configure 
3. Select TCP/IP Configuration
4. Select vMotion > Edit
5. Under Routing, enter the Default Gateway in the VMKernel Gateway section
6. Run vmkping -I vmkX #.#.#.# -S vmotion from the source, pinging the destination host's vMotion IP address
7. Run vmkping -I vmkX #.#.#.# -S vmotion from the destination, pinging the source host's vMotion IP address
8. Attempt the vMotion again
   

 

If the vMotion VMkernel Adapter is using the Default TCP/IP stack: 

1. Select the host in question
2. Select Configure
3. Select VMkernel adapters
4. Edit Settings of the vMotion VMkernel adapter
5. Select IPv4 settings
6. Check the "Override default gateway for this adapter" box
7. Select OK
8. Attempt the vMotion again