Understand vMotion networking requirements

Products

• VMware vSphere ESXi

Issue/Introduction

This article will help us to understand the requirements of vMotion networking.

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0

Resolution

vMotion Data Flow and Network Requirements
vMotion transfers two types of data, cold data and hot data. Cold data refers to VM data that does not change during vMotion, such as VM snapshots, non child most delta disks, and vmx logs. Hot data refers to VM data that changes during vMotion, such as memory state, device state, and child most delta disks.

VMKernel Port for Cold Data Transfer
If any vmkernel port is configured for provisioning traffic, either through setting up a provisioning netstack, or simply enable provisioning traffic on a particular port, cold data would traverse through this port. If no vmkernel port is configured for provisioning traffic, cold data would traverse through the management vmkernel port.

VMKernel Port for Hot Data Transfer
If any vmkernel port is configured for vMotion traffic, either through setting up a vMotion netstack, or simply enable vMotion traffic on a particular port, hot data would traverse through this port. Some vmkernel port must be configured for vMotion traffic in order to perform vMotion.

Requirements for VMKernel Ports Used in vMotion Data Transfer
A successful Long Distance vMotion requires successfully transferring both cold and hot data. Therefore, depending on the which ports was used (provisioning + vMotion or management + vMotion), the traffic must be routable between source and destination hosts on these ports. Both L2 and L3 is supported for VMKernel Ports used for vMotion (cold and hot data transfer), as long as there is connectivity.

Requirements for VM Network
VM Network is the network that services network traffic to and from a VM. Only L2 is supported for VM Network.

For more information see vSphere vMotion Networking Requirements.

Long Distance vMotion

Definition
Long Distance vMotion is vMotion with latency between hosts greater than 4 milliseconds. This means all the requirements and features that applies to vMotion also applies to Long Distance vMotion.

Requirements
In addition to the basic vMotion network requirements outlined above, your environment must also comply with the following requirements:

1. A RTT (round-trip time) latency of 150 milliseconds or less, between hosts.
2. Your license must cover vMotion across long distances. The long distance vMotion features require an Enterprise Plus license. For more information, see Compare vSphere Editions.

Cross vCenter vMotion
For more information see Cross vCenter Migration and Clone requirements in VMware vSphere 6.0 and later.

Encrypted vMotion
For more information see Encrypted vSphere vMotion.

Frequently Asked Questions

1. Is Long Distance vMotion traffic encrypted?

 

It depends. As mentioned in previous section, there are two types of data that vMotion transfers, each with its own transfer protocol. For cold data, NFC (Network File Copy) is used, and it uses TLS by default, hence cold data is already encrypted, regardless of the underlying network. For hot data, vMotion protocol is used, and whether this traffic is encrypted or not depends on whether Encrypted vMotion is used. To achieve secure data transfer for hot data without Encrypted vMotion, we recommend either have an isolated vMotion network, or use IPSec. Long Distance vMotion follows the exact same guidelines as vMotion outlined above, in terms of traffic security.

2. Does Long Distance vMotion require Cross vCenter vMotion? And does the vCenters need to be on the same SSO domain?

Long Distance vMotion can be performed regardless of the number of vCenters or same/different SSO domains. The only thing distinguishes Long Distance vMotion from vMotion is the latency between source and destination hosts. You may be subject to additional requirements if you were to perform Cross vCenter vMotion, or Cross vCenter vMotion with Enhanced Linked Mode, but those requirements are orthogonal to Long Distance vMotion.

3. Is provisioning network required for Long Distance vMotion?

No, same as vMotion, if no provisioning network is configured, the management network will be used for cold data transfer. Only vMotion network is required, for hot data transfer.

4. How to tune Long Distance vMotion bandwidth for better performance (for hot data)?

By default, Long Distance vMotion operates on an expected line rate of 1Gbps, and adjust socket buffer size based on this expected line rate. If you environment supports higher line rate, you can change this default expected line rate to your actual line rate by setting the following VSISH option:

vsish -e set config/Migrate/intOpts/NetExpectedLineRateMBps <expected line rate>

For vSphere 6.7u1, you will also be able to increase the maximum socket buffer size by setting the following VSISH option:

vsish -e set /net/tcpip/instances/<netstack>/sbMax <max ideal socket buffer size>

<netstack> should either be the default networking stack, or the vMotion networking stack, depends on which one you were using for vMotion hot data traffic. <max ideal socket buffer size> should be the maximum bandwidth delay product you expect in your environment.