Reduced Throughput when using Overlay Segments Between Hosts

Products

• VMware NSX-T Data Center
• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

Reduced throughput when using overlay segments between hosts


Only one poll world CPU maybe in use. We can check this by collecting the netstats output -

ESXi command to generate the below information:

# net-stats -A -t WwQqihVvh -i 10 > "/tmp/netstats-"$HOSTNAME".out" 2>/dev/null

Output found in /tmp/netstats-"$HOSTNAME".out:

"2102330": {"id": 2102330, "used": 0.05, "ready": 0.02, "cstp": 0.00, "name":
"vmnic4-pollWorld-0-0x43025413ce80",
"2102331": {"id": 2102331, "used": 99.81, "ready": 0.00, "cstp": 0.00, "name":
"vmnic4-pollWorld-1-0x43025413cfc0", <-Only pollWorld is showing % used
"2102332": {"id": 2102332, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name":
"vmnic4-pollWorld-2-0x43025413aec0",
"2102333": {"id": 2102333, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name":
"vmnic4-pollWorld-3-0x43025413b000",
"2102334": {"id": 2102334, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name":
"vmnic4-pollWorld-4-0x430254086840",
"2102335": {"id": 2102335, "used": 0.00, "ready": 0.00, "cstp": 0.00, "name":
"vmnic4-pollWorld-5-0x430254087340",

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center 4.x

Cause

RSS is not enabled inside the guest OS.

To verify if RSS is enabled on vmxnet3 or not, run the following command from Windows Command prompt -

# Get NetAdapterRss -Name <adapter_name>

From Windows GUI, we can enable RSS :

Device manager >> Network Adapters >> vmxnet3 Ethernet Adapter >> Properties >> Advanced

Select Receive Side Scaling property and ensure that it is enabled.

Additionally, you can also run netsh int tcp show global on the command prompt to verify if RSS is enabled or not for Windows guest VMs.



In the case of Linux, run :

ethtool -x <ethID>

If the RSS hash table has all 0s, that means that RSS is disabled.

Eg - In this case, since the indirection table has non-zero values, it means that RSS is enabled.

root@ubuntu:~$ ethtool -x ens34
RX flow hash indirection table for ens34 with 4 RX ring(s):
    0:      0     1     2     3     0     1     2     3
    8:      0     1     2     3     0     1     2     3
   16:      0     1     2     3     0     1     2     3
   24:      0     1     2     3     0     1     2     3
RSS hash key:
Operation not supported
RSS hash function:
    toeplitz: on
    xor: off
    crc32: off

Resolution

This is a misconfiguration. We can correct it by enabling RSS and LRO.

Workaround:
Add ethernetX.pnicFeatures = "4" (RSS) or "5" (LRO+RSS) to the vNIC's in the .vmx file and check if the throughput performance increases.

To do so, follow - https://knowledge.broadcom.com/external/article/311480