Alarm for high or very high Enhanced Datapath flow table usage in VMware NSX

Products

• VMware NSX

Issue/Introduction

• An alarm is showing in the UI that states either "Alarm for high EDP flow table usage" or "Alarm for very high EDP flow table usage".
  
  Note: This alarm fires as soon as any individual thread reaches the threshold; it does not mean all threads on the host are at capacity.

Environment

• VMware NSX 4.2.1 through 4.2.4
• VMware NSX 9.0.0 through 9.1.0
• VMware Cloud Foundation

Cause

• This elevated usage typically occurs when any thread in the system processes a high volume of short-lived connections, causing active flows to approach the current flow-table allocation limit for that thread.
• If the number of active flows exceeds the flow-table size, or if there are many short-lived connections coming in, the flow tables can fill up.
• The alarm did not exist prior to version 4.2.0, so the table usage may still have been high and would not have been reported prior to 4.2.0.

Note: Even though the alarm was introduced in NSX 4.2.0, it will not trigger in VCF 5.2 (NSX 4.2) environments if Enhanced Data Path (EDP) standard mode is not enabled on the hosts.

Resolution

• These alerts are generated as Warnings in the system. However, they are intended to be interpreted as Informational only, and cany typically be safely suppressed.

• The Enhanced Datapath (EDP) utilizes multiple tables and a dynamic allocation process to mitigate these scenarios automatically. This architecture ensures that standard application traffic remains unaffected and the system maintains stable operation without experiencing thrashing behavior.

• This alarm is not present in newer versions of NSX (9.1.1)

If performance degradation is observed in conjunction with this alarm in NSX versions 4.2.1 through 4.2.4, or 9.0.0 through 9.1.0:

• Consider increasing the flow table size by invoking the following command:
  1. Analyze current flow count: Run the following command on the affected host to determine how many flows are currently created:

     nsxdp-cli ens flow-table dump -s 0 | wc -l

     Note: Run this command multiple times throughout the day to calculate an average baseline.
     Get the current size of the table (Default value is 131,072):

     nsxdp-cli ens flow-table size get

  2. Increase flow table size: If the active flow count consistently exceeds the default (131,072) and you confirm stable high-volume traffic, increase the size:

     nsxdp-cli ens flow-table size set -s $NUM_ENTRIES

     • $NUM_ENTRIES must be a power of 2 (max value 524,288).
     • The host must be rebooted for changes to take effect.

Note:

• The value set is persistent across reboots from NSX version 4.2.1.
• Increasing the number of flow entries doesn't always improve performance, if short lived connections keep coming in. The flow table might always be full regardless of the flow table size. A large flow-table size wouldn't help in this case. EDP has a logic to detect this and automatically enable and disable flow tables to handle such a case.
• Increasing the number of flow entries may increase the memory footprint.
• If no actual performance issues (latency/packet loss) are reported, this alarm can often be treated as cosmetic.

Additional Information

Flow tables being full does not indicate any impact, just that performance improvements from EDP are reduced. For this reason, this alarm will be a lower warning level in future releases of NSX. 

API Guide
Admin Guide
Increasing the flow table size on ESXi node does not persist after a reboot

Increasing the flow table size is beneficial when the number of active flows is stable over time. However, this might not improve performance in cases where short-lived connections continuously come in. In such scenarios, the flow table might always be full, regardless of its size.