Increased latency observed for applications and NSX Edges
Article ID: 374777
Updated On:
Products
Issue/Introduction
Increased application latency is observed when application traffic is using NSX overlay networks that use Geneve encapsulation and the ESXi hosts are using QLogic FastLinQ QL45xxx as the NIC
Environment
VMware vSphere ESXi
VMware NSX-T 3.x
VMware NSX 4.x
Cause
When the RXPPS on this queue is high enough, the netpoll world that responsible for handling the rx traffic will be saturated. As qedentv driver tx/rx queue is paired, the saturated netpoll world is unable to quickly complete tx packets for the paired default txq. As the result we can observe the tx latency, when the tx packet is sent by the default txq.
Resolution
The solution is to enable DRSS in the qedentv driver, which automatically spreads the rx traffic from the default rxq to the DRSS queue pool, eliminates the I/O bottleneck on the single netpoll world. DRSS comes disabled by default in the qedentv driver due to a historical issue FJT-9147 (manufacturer issue) which caused Geneve tunnels to go down if DRSS was enabled. This issue was fixed in the following driver versions of the qedentv driver (refer to the Release Notes for the driver that are included in the driver zip file):
ESXi 7.0 : Version 3.70.50.0
ESXi 8.0 : Version 3.71.50.0
First, ensure that the qedentv driver is higher than the versions mentioned above, but it's recommended to have your driver versions higher than the following, to avoid some other DRSS related known issues too:
ESXi 7.0 : Version 3.70.52.0
ESXi 8.0 : Version 3.71.52.0
Additionally, the qedentv firmware (MFW) should be updated to the latest compatible version too, to maintain compatibility with the updated driver.
Once the driver and firmware versions are updated to the latest approved versions, please run the following command on every ESXi host to enable DRSS and then reboot the ESXi host for the changes to take effect:
esxcli system module parameters set -p "DRSS=6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6" -m qedentv
Additional Information
Feedback
