"vMotion failed because the destination host did not receive data from the source host on the vMotion network" - Cross vCenter vMotion fails at 45%

Products

• VMware vCenter Server

Issue/Introduction

• On the vSphere Client cross vCenter vMotion fails at 45% with the error

The vMotion failed because the destination host did not receive data from the source host on the vMotion network. Please check your vMotion network settings and physical network configuration and ensure they are correct.
failed to connect to remote host <IP address> from host <IP address>: Network unreachable
vMotion migration ### failed to create a connection with remote host <IP address> 
The ESX hosts failed to connect over the VMotion network 
The vMotion migrations failed because the ESX hosts were not able to connect over the vMotion network

• In the source ESXi log, /var/run/log/hostd.log, a similar message to this example is seen
  
  <DATE>T15:08:52.150Z Db(#) Hostd[###]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/<datastore>/<VM name>.vmx] VMotionInitiateSrc: Start message: The vMotion migrations failed because the ESX hosts were not able to connect over the vMotion network.  Check the vMotion network settings and physical network configuration.
<DATE>T15:08:52.150Z Db(#) Hostd[###]: --> VMotionStream [<ID>] failed to create a connection with remote host <IP address>: The ESX hosts failed to connect over the VMotion network
<DATE>T15:08:52.150Z Db(#) Hostd[###]: --> Migration [<ID>] failed to connect to remote host <IP address> from host <IP address>: Network unreachable.
  
  
• Cold migration of the VM works successfully
• The target host vMotion vmkernel is on a different subnet to the source host vMotion vmkernel interface
• The vmkernel configured for vMotion is:
  • on the default network stack
  • is an internal network with no gateway
  • is on a different subnet to the management vmkernel
• vMotion to hosts which have vMotion vmkernel in the same subnet on the source and destination host succeed

Environment

VMware vCenter 8.x

Cause

vMotion requires port 902 IP connectivity between management vmkernel interfaces and port 8000 IP connectivity between vMotion interfaces.
When using the default network stack the vMotion vmkernel can route to another network through the management default gateway only when it is configured on the same subnet. In this case the vMotion interface cannot connect with a vMotion interface on a different subnet resulting in vMotion failure.

Resolution

Reconfiguration is required

• Reconfigure the vMotion vmkernel to have an IP address on the same subnet as the management vmkernel. Make this change on both sides. This will allow vMotion traffic to route over the management vmkernel

or

• On physical create a default gateway for the vMotion network and configure a vMotion network stack on the host with this default gateway