"No Data to show" for some NSX Distributed Firewall flows due to EXPIRE_OUTBOUND limit in VCF Operations for Networks

Products

• VCF Operations for Networks

Issue/Introduction

VCF Operations for Networks displays "No Data to show" for certain protected NSX Distributed Firewall (DFW) flows. Specific network flows fail to display the expected associated firewall rule ID in the GUI. This occurs despite VMware NSX being successfully configured as a data source and security group memberships functioning normally.

When querying flows for some IP addresses but not others, zero results are displayed in the GUI.

 

Upon inspection of the Collector flow processor logs (/var/log/arkin/flow-processor), you find high EXPIRE_OUTBOUND rejection statistics, for example:

INFO v2.state.FiveTupleCaches NFCAPD_vds printAndResetRejectionStats:202 Stateful Rejection Stats: SHARD-0 <EXPIRE_INCOMPLETE_TCP:0 EXPIRE_OUTBOUND:0 DROP_INTERNET_DENY:0> SHARD-1 <EXPIRE_INCOMPLETE_TCP:0 EXPIRE_OUTBOUND:0 DROP_INTERNET_DENY:0> SHARD-2 <EXPIRE_INCOMPLETE_TCP:7323 EXPIRE_OUTBOUND:24693 DROP_INTERNET_DENY:0>

 

You may see that affected VMs doing port scanning that can create lot of connections. This can been identified from the packet capture if there is a long list of ports sending flows.

 

NOTE: VCF Operations for Networks was formerly named Aria Operations for Networks (AON), and prior to that was named vRealize Network Insight (vRNI).

Environment

VMware Aria Operations for Networks

VMware NSX

Cause

The Aria Operations for Networks Collector is dropping flows due to the EXPIRE_OUTBOUND connection limit being exceeded.

When a source IP generates an excessive number of outbound connections (such as during a port scan) that match a DROP rule, the DFW flows exceed the hardcoded default limit of 50 flows per client IP. Consequently, the excess flows are discarded before they are processed and stored in VCF Operations for Networks.

Resolution

1. Identify and stop any port scanning or excessive connection-generating activity on the affected source virtual machine.
   
   
2. Change the IP address of the affected VM to an unused IP address in order to immediately validate flow processing functionality. Otherwise, you must wait for at least 25 hours to bypass the existing connection cache.

Terminating the excessive connection attempts keeps the connection count below the threshold of 50. Because the VCF Operations for Networks cache retains connection limits for a specific duration, modifying the test IP bypasses the existing cache, allowing immediate validation of correctly stitched DFW flows in the GUI.