High Latency observed in NSX-Prepared Clusters when using Enhanced Data Path (EDP)

Products

• VMware NSX

Issue/Introduction

• Applications experience increased latency when the traffic traverses NSX-prepared hosts with Enhanced Data Path (EDP) enabled.
• Latency is most noticeable for ultra-latency-sensitive applications, potentially resulting in:
      1. Application timeouts
      2. Job failures
      3.Degraded performance trough virtual network application (e.g., F5)
• No packet loss or health alarms are reported in NSX Manager.
• Issue is reproducible when workloads or appliances run on hosts with EDP enabled and improves or disappears when EDP is disabled.

Environment

VMware NSX 4.x

VMware vSphere ESXi

Cause

Due to a bug in the driver, an ENS NetWorld thread is consuming 100% of available CPU, even without any network load. 

Resolution

• Upgrade to NSX version 4.2.3.2 or higher. 
• ESXi 8.0.3 Patch 07
• QLogic NIC driver version 3.71.9.0

Additional Information

Please see detailed explanation:

The Problem: A "False Alarm" Loop.

The system uses a signaling unit called the IGU to decide when to notify the computer that new data has arrived. It works like a simple balance scale:
The Producer: Tells the system how much data has been sent.
The Consumer: Tells the system how much data has been processed.
The Rule: If these two numbers are equal, everything is caught up and the system stays quiet. If they are different, an interrupt (a notification) is triggered.

What goes wrong:
During startup, the system resets the "Consumer" counter to zero. However, the "Producer" counter (hidden in the hardware firmware) might not be zero. Because they don't match, the system thinks there is new data to process and constantly triggers notifications—even though there are no actual packets. This continues until a real piece of data arrives and forces both numbers to sync back up.