VMXNET3 Virtual Network Interface Adapter Only Utilizes Single Receive Queue

Products

• VMware NSX
• VMware vSphere ESXi

Issue/Introduction

Multiple guest applications experience elevated network latency, packet loss, or unexpected disconnections across logical switches. Inter-VM overlay and UDP traffic processes exclusively on a single VMXNET3 receive (Rx) queue despite multiple queues being allocated.

Execution of the net-stats utility outputs single-queue utilization similar to the following:

{
  "name": "<REDACTED_HOSTNAME>.eth1",
  "switch": "DvsPortset-1",
  "id": 67108875,
  "mac": "<REDACTED_MAC>",
  "rxmode": 0,
  "tunemode": 0,
  "uplink": "false",
  "ens": false,
  "txpps": 168674,
  "txmbps": 1006.2,
  "txsize": 745,
  "txeps": 0.00,
  "rxpps": 176834,
  "rxmbps": 1054.9,
  "rxsize": 745,
  "rxeps": 0.00,
  "vnic": {
    "type": "vmxnet3",
    "ring1sz": 4096,
    "ring2sz": 128,
    "tsopct": 0.0,
    "tsotputpct": 0.0,
    "txucastpct": 45.0,
    "txeps": 0.0,
    "lropct": 0.0,
    "lrotputpct": 0.0,
    "rxucastpct": 100.0,
    "rxeps": 0.0,
    "maxqueuelen": 0,
    "requeuecnt": 0.0,
    "agingdrpcnt": 0.0,
    "deliveredByBurstQ": 0.0,
    "dropsByBurstQ": 0.0,
    "txdisc": 0.0,
    "qstop": 0.0,
    "txallocerr": 0.0,
    "txtsosplit": 0.0,
    "r1full": 0.0,
    "r2full": 0.0,
    "sgerr": 0.0
  },
  "rxqueue": {
    "count": 4,
    "details": [
      {"intridx": 0, "pps": 176834, "mbps": 1054.9, "errs": 0.0},
      {"intridx": 1, "pps": 0, "mbps": 0.0, "errs": 0.0},
      {"intridx": 2, "pps": 0, "mbps": 0.0, "errs": 0.0},
      {"intridx": 3, "pps": 0, "mbps": 0.0, "errs": 0.0}
    ]
  },
  "txqueue": {
    "count": 4,
    "details": [
      {"intridx": 0, "pps": 41623, "mbps": 244.6, "errs": 0.0},
      {"intridx": 1, "pps": 45065, "mbps": 272.6, "errs": 0.0},
      {"intridx": 2, "pps": 48082, "mbps": 296.3, "errs": 0.0},
      {"intridx": 3, "pps": 41454, "mbps": 237.1, "errs": 0.0}
    ]
  }
}

Environment

 

• VMware vSphere ESXi 7.0.x

• VMware vSphere ESXi 8.0.x

• VMware NSX-T Data Center 3.x

• VMware NSX 4.x

 

Cause

Receive Side Scaling (RSS) for UDP/Overlay traffic is disabled or unconfigured within the virtual machine advanced settings (.vmx), causing all inbound UDP packets to route through virtual interrupt index 0 (intridx: 0).

Resolution

1. Power off the affected virtual machine.

2. Edit the .vmx configuration file using the vSphere Client or ESXi CLI.

3. Append or modify the following configuration options:

   • ethernetX.pnicFeatures = "4"

   • ethernetX.udpRSS = "1" (Replace X with the target network adapter index number, e.g., ethernet0)

4. Save the .vmx file and power on the virtual machine.

 Where, X needs to be replaced by the number of virtual network card to which the feature should be added.

Note:  If these configurations are not present in the .vmx file, add them using the steps in this doc:   Editing the .vmx file of a VMware Workstation and VMware Player virtual machine

 

Additional Information

 

• Configure Receive-Side Scaling (RSS)

• Editing the .vmx file of a Virtual Machine