Improving performance for vMotion when using a 40GbE NIC

Products

• VMware vCenter Server

Issue/Introduction

To get the optimal vMotion throughput on a 40 gigabit Ethernet network adapter (40GbE NIC), it is recommended you configure multiple vMotion-enabled vmknics per single physical 40GbE NIC. During the initial vMotion setup phase, vMotion in vSphere 6 creates TCP/IP connections (or stream channels) between the source and destination hosts based on the configured vMotion vmknics, and then instantiates one vMotion worker thread per TCP/IP connection. When using a single vMotion vmknic with a 40GbE NIC, the peak vMotion bandwidth is limited to 18 gigabits per second because the vMotion worker thread becomes CPU saturated.

Environment

VMware vCenter Server 7.0

VMware vCenter Server 8.0

Resolution

To achieve the line rate on a 40GbE vMotion network, it is recommended you configure three vMotion vmknics on a 40GbE NIC. Using multiple vMotion vmknics enables vMotion to instantiate multiple vMotion worker threads, thereby use more CPU resources.

Performance test results show how altering the number of vMotion vmknics increases the performance of vMotion when using a 40GbE NIC.

Number of vMotion vmknics	Number of 40GbE NICs	Throughput
1	1	18Gb per second
2	1	28Gb per second
3	1	36Gb per second

Additional Information

NIC limits for concurrent operations

• A 1Gb NIC has a capacity of 4 units, so you can do 4 vMotion operations at a time from a given 1Gb NIC.
• A 10Gb NIC has a capacity of 8 units, so you can do 8 vMotion operations at a time from a given 10Gb NIC.
• A 25Gb NIC has as capacity of 8 units, so you can do 8 vMotion operations at a time from a given 25Gb NIC.