Alarm for high or very high Enhanced Datapath flow table usage in NSX.

Products

• VMware NSX

Issue/Introduction

• An alarm is showing in the UI that states either "Alarm for high EDP flow table usage" or "Alarm for very high EDP flow table usage".

 

Environment

• VMware NSX 4.2.x

Cause

• If the number of active flows exceeds the flow-table size, or if there are many short-lived connections coming in, the flow tables can fill up.
• The alarm did not exist prior to version 4.2.0, so the table usage may still have been high and would not have been reported prior to 4.2.0.
• Note that even though the alarm was introduced in NSX 4.2.0, it will not trigger in VCF 5.2 (NSX 4.2) environments if Enhanced Data Path (EDP) standard mode is not enabled on the hosts.

Resolution

• For Enhanced Datapath (EDP) host switch mode, for versions 4.2.1 and above, if performance degradation is observed consider increasing the flow table size by invoking the following command:
  1. Analyze current flow count: Run the following command on the affected host to determine how many flows are currently created:
     -  nsxdp-cli ens flow-table dump -s 0 | wc -l (Note: Run this command multiple times throughout the day to calculate an average baseline.)
     
     Get the current size of the table: (Default value is 131,072)
     - nsxdp-cli ens flow-table size get

  2. Increase flow table size: If the active flow count consistently exceeds the default (131,072) and you confirm stable high-volume traffic, increase the size using:
     - nsxdp-cli ens flow-table size set -s $NUM_ENTRIES

     • $NUM_ENTRIES must be a power of 2 (max value 524,288).

     • The host must be rebooted for changes to take effect.

• Note:
  • The value set is persistent across reboots from NSX version 4.2.1.
  • Increasing the number of flow entries doesn't always improve performance, if short lived connections keep coming in. The flow table might always be full regardless of the flow table size. A large flow-table size wouldn't help in this case. EDP has a logic to detect this and automatically enable and disable flow tables to handle such a case.
  • Increasing the number of flow entries may increase the memory footprint.
  • If no actual performance issues (latency/packet loss) are reported, this alarm can often be treated as cosmetic.

Additional Information

Flow tables being full does not indicate any impact, just that performance improvements from EDP are reduced. For this reason, this alarm will be a lower warning level in future releases of NSX. 

API Guide
Admin Guide
Increasing the flow table size on ESXi node does not persist after a reboot

Increasing the flow table size is beneficial when the number of active flows is stable over time. However, this might not improve performance in cases where short-lived connections continuously come in. In such scenarios, the flow table might always be full, regardless of its size.