Vnic Using Only One of Multiple Available Rxqueues
Article ID: 317212
Updated On:
Products
VMware NSX
VMware vSphere ESXi
Issue/Introduction
Symptoms:
Multiple production applications are experiencing performance issues and disconnecting. Latency and packet loss can be seen between VMs on different logical switches.
All the overlay, UDP traffic will hit the same VMXNET3 rx queue.
ESXi command to generate the below information:
# net-stats -A -t WwQqihVvh -i 10 > "/tmp/netstats-"$HOSTNAME".out" 2>/dev/null
The output of this command, found in /tmp/netstats-<hostname>.out.
Inside this file, find the virtual machine's name and the ethernet port processing applicable traffic.
{"name": "<virtual-machine-name>.eth1", "switch": "DvsPortset-1", "id": 67108875, "mac": "##:##:##:##:##:91", "rxmode": 0, "tunemode": 0, "uplink": "false"
, "ens": false,
"txpps": 168674, "txmbps": 1006.2, "txsize": 745, "txeps": 0.00, "rxpps": 176834, "rxmbps": 1054.9, "rxsize": 745, "rxeps": 0.00,
"vnic": { "type": "vmxnet3", "ring1sz": 4096, "ring2sz": 128, "tsopct": 0.0, "tsotputpct": 0.0, "txucastpct": 45.0, "txeps": 0.0,
"lropct": 0.0, "lrotputpct": 0.0, "rxucastpct": 100.0, "rxeps": 0.0,
"maxqueuelen": 0, "requeuecnt": 0.0, "agingdrpcnt": 0.0, "deliveredByBurstQ": 0.0, "dropsByBurstQ": 0.0,
"txdisc": 0.0, "qstop": 0.0, "txallocerr": 0.0, "txtsosplit": 0.0, "r1full": 0.0, "r2full": 0.0, "sgerr": 0.0},
"rxqueue": { "count": 4, "details": [
{"intridx": 0, "pps": 176834, "mbps": 1054.9, "errs": 0.0},--- Only rx 0 has traffic
{"intridx": 1, "pps": 0, "mbps": 0.0, "errs": 0.0},
{"intridx": 2, "pps": 0, "mbps": 0.0, "errs": 0.0},
{"intridx": 3, "pps": 0, "mbps": 0.0, "errs": 0.0} ]},
"txqueue": { "count": 4, "details": [
{"intridx": 0, "pps": 41623, "mbps": 244.6, "errs": 0.0},
{"intridx": 1, "pps": 45065, "mbps": 272.6, "errs": 0.0},
{"intridx": 2, "pps": 48082, "mbps": 296.3, "errs": 0.0},
{"intridx": 3, "pps": 41454, "mbps": 237.1, "errs": 0.0} ]}
, "ens": false,
"txpps": 168674, "txmbps": 1006.2, "txsize": 745, "txeps": 0.00, "rxpps": 176834, "rxmbps": 1054.9, "rxsize": 745, "rxeps": 0.00,
"vnic": { "type": "vmxnet3", "ring1sz": 4096, "ring2sz": 128, "tsopct": 0.0, "tsotputpct": 0.0, "txucastpct": 45.0, "txeps": 0.0,
"lropct": 0.0, "lrotputpct": 0.0, "rxucastpct": 100.0, "rxeps": 0.0,
"maxqueuelen": 0, "requeuecnt": 0.0, "agingdrpcnt": 0.0, "deliveredByBurstQ": 0.0, "dropsByBurstQ": 0.0,
"txdisc": 0.0, "qstop": 0.0, "txallocerr": 0.0, "txtsosplit": 0.0, "r1full": 0.0, "r2full": 0.0, "sgerr": 0.0},
"rxqueue": { "count": 4, "details": [
{"intridx": 0, "pps": 176834, "mbps": 1054.9, "errs": 0.0},--- Only rx 0 has traffic
{"intridx": 1, "pps": 0, "mbps": 0.0, "errs": 0.0},
{"intridx": 2, "pps": 0, "mbps": 0.0, "errs": 0.0},
{"intridx": 3, "pps": 0, "mbps": 0.0, "errs": 0.0} ]},
"txqueue": { "count": 4, "details": [
{"intridx": 0, "pps": 41623, "mbps": 244.6, "errs": 0.0},
{"intridx": 1, "pps": 45065, "mbps": 272.6, "errs": 0.0},
{"intridx": 2, "pps": 48082, "mbps": 296.3, "errs": 0.0},
{"intridx": 3, "pps": 41454, "mbps": 237.1, "errs": 0.0} ]}
Environment
VMware vSphere ESXi 7.0.x
VMware NSX-T Data Center 3.x
VMware vSphere ESXi 8.0.x
VMware NSX-T Data Center 4.x
VMware NSX-T Data Center
VMware NSX-T Data Center 3.x
VMware vSphere ESXi 8.0.x
VMware NSX-T Data Center 4.x
VMware NSX-T Data Center
Resolution
Modify the virtual machine's .vmx file configuration with the settings denoted in the Workaround section.
Workaround:
This could be a case of misconfiguration. RSS may not be enabled. Check if the configurations mentioned below are present in the .vmx file of the affected VM. If not, make sure to edit the vmx file and add these configurations.
Where, X needs to be replaced by the number of virtual network card to which the feature should be added.
This ensures that the rx packets get consumed evenly over all the rx queues.
If these configurations are not present in the .vmx file, add them using the steps in this doc:
Workaround:
This could be a case of misconfiguration. RSS may not be enabled. Check if the configurations mentioned below are present in the .vmx file of the affected VM. If not, make sure to edit the vmx file and add these configurations.
ethernetX.pnicFeatures="5"
ethernetX.updRSS="1"
ethernetX.updRSS="1"
Where, X needs to be replaced by the number of virtual network card to which the feature should be added.
This ensures that the rx packets get consumed evenly over all the rx queues.
If these configurations are not present in the .vmx file, add them using the steps in this doc:
Feedback
Yes
No
Powered by
