Examining basic Long-Distance vMotion (LDVM) ports and flows

Products

• VMware vSphere ESXi

Issue/Introduction

Long Distance vMotion (LDVM) is a feature of vSphere 6.x which allows a running workload to be moved from one vCenter instance to another, within the same Platform Services Controller (PSC) single-sign-on (SSO) domain. LDVM inherently involves the transfer of disk and memory state, typically from one datacenter to another. This is often desired at greater than campus distances. At a mechanical level, this means "moving" the running VM instance from a sending ESXi instance to a receiving ESXi instance.

Environment

VMware vSphere 6.x

Resolution

The LDVM is effected by the transfer of two distinct kinds of data. These flows may use two different communications paths.
The memory state of the running VM is transferred from the sending ESXi hypervisor to the receiving ESXi hypervisor over the vMotion vmkernel port. In a given ESXi implementation, this may or may not be the default vmkernel port, as the binding of vMotion is configurable.
The disk files of the running VM are transferred (copied, then deleted) from the sending ESXi hypervisor to the receiving ESXi hypervisor over the vSphereProvisioning vmkernel port. In a given ESXi implementation, this may or may not be the default vmkernel port, as the binding of vSphereProvisioning is configurable.

When LDVM does not succeed, troubleshoot that the required ports communicate properly...

From a command line on a given ESXi host, diagnose that basic routing is correctly configured on the network between ESXi hosts. Exploit "network diag" (the successor to vmkping) between ESXi-Seattle and ESXi-Dallas, for example:

esxcli network diag ping -I vmk0 --netstack=defaultTcpipStack-H #.#.20.#
In this case, the default TCP/IP stack is used for Management of the ESXi host, and on a sample IP address the third octet of which is 20. Use the real IP address of the target ESXi host. Perform this test in both directions.

esxcli network diag ping -I vmk1 --netstack=vmotion -H #.#.30.#
In this case, the vmotion stack is used for vMotion, and on a sample IP address the third octet of which is 30. Use the real IP address of the target ESXi host. Perform this test in both directions.
This netstack is used to transfer the memory state of the running VM.

esxcli network diag ping -I vmk2 --netstack=vSphereProvisioning -H #.#.40.#
In this case, the vSphereProvisioning stack is used so-called provisioning of VMs to the ESXi host, and on a sample IP address the third octet of which is 40. Use the real IP address of the target ESXi host. Perform this test in both directions.
This netstack is used to transfer the disk files of the running VM.

Workaround:
VMs located on datastores other than internal storage (including Fibre Channel, iSCSI, NFS, or a vMSC stretched cluster) will fail to copy their VM disks over a non-default vSphereProvisioning vmkernel port using NFC, regardless of hypervisor configuration. As such, in any hardened architectures with VLANs that are air-gapped by purpose, a long-distance vMotion (LDVM) may fail over non-default vmkernel ports.

This is a known issue affecting vCenter Server 6.0.x and 6.5.x versions.

Currently, there is no resolution.