Increased application latency detected in an NSX Environment on an NSX-T prepared host

Products

• VMware NSX

Issue/Introduction

• ESXi host is prepared for NSX
• Applications may experience heightened latency when traversing both host and Edge TEPs
• Hardware LRO is enabled on the ESXi host's NICs
• Geneve Offload is enabled on the ESXi host's NICs

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4.x

Cause

Most NICs do not support or do not properly support LRO for Geneve traffic.  NICs using the bnxtnet and qlogic drivers do not properly report to ESXi that LRO for Geneve traffic will not work.  This leads to overhead on the software stack that degrades network performance.

Resolution

In order of preference:

1. Upgrade to ESXi 7.0u3EP10 or 8.0U1.  These ESXi versions will bypass LRO for Geneve traffic.
2. Disable LRO using the NIC driver config parameter.  For bnxtnet and qedentv, it’s via the disable_tpa parameter.
    

Additional Information

Network cards used for VSAN storage traffic should keep Hardware LRO enabled if it is supported.

Impact/Risks:
For NICs processing GENEVE traffic, with hardware LRO enabled, applications may see increased latency or other signs of degraded performance.