High virtual network throughput performance tuning recommendation when using 100G network interface cards

Products

• VMware vSphere ESXi
• VMware NSX

Issue/Introduction

• This article provides performance tuning recommendations for physical NICs in a vSphere environment.
• Users might not see a high enough throughput for a VNIC to fill the physical NIC bandwidth.

Environment

• VMware NSX-T
• VMware NSX 4.0.0.1
• VMware vSphere ESXi 8.0.2

Cause

• By default, ESX/NSX networking uses limited number of threads to service Tx and Rx of a vNIC. Throughput of a vNIC is limited by the processing capacity of those threads.

Resolution

• If a vNIC requires high throughput, whether just high Rx side throughput or bi-directional, it is recommended to configure ethernetX.pnicFeatures=4, and ethernetX.ctxPerDev=3 in the virtual machine's VMX file, to get the parallelism in the networking layer. This gives a much more balanced Tx side and Rx side bandwidth, roughly equally in the bi-directional test. These tunings work best with pNIC RSS enabled.

Relevant documents:
Receive Side Scaling
Performance Best Practices for VMware vSphere 8.0

Additional Information

Enabling RSS, and other tuning on the Guests vNICs may cause higher CPU requirements for the edited VM, possibly affecting other VMs negatively by lowering the CPU resources available.