SYN,ACK packets hitting unexpected DFW rule during TCP/SYN scan

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

• You have DFW rules applied to a VM that is generating many TCP connections (e.g. a SYN scanner).
• During a SYN scan, SYN,ACK packets can be seen hitting an unexpected DFW rule in the dfwpktlogs.log on the ESXi host.
• From the dfwpktlogs.log on the ESXi host the Scanner VM resides on, you can see that flows are timing out earlier than expected
• In the following example, the connection was timed out after only 2 seconds: 

4b84#### INET match PASS 3072 OUT 48 TCP 10.1.29.22/7641->10.2.201.207/111 S                  <---- Initial SYN packet hits expected rule. This creates the flow table entry
4b84#### INET TERM PASS 3072 OUT TCP TIMEOUT 10.1.29.22/7641->10.2.201.207/111 1/1 48/44      <---- The TIMEOUT indicates that the flow is timed out and removed from the flow table
4b84#### INET match PASS 15402 IN 44 TCP 10.2.201.207/111->10.1.29.22/7641 SA                 <---- The next SYN,ACK packet does not match the previous flow since it is already removed from the flow table

Environment

VMware NSX - All Versions

Cause

• TCP or SYN scans generate a very high rate of new connections.

• This triggers the DFW Adaptive Timeout, which dynamically reduces connection timeouts as the number of flows grows.

• Default thresholds (per vnic per rule):

  • dfw.adaptive_start = 200,000 connections

  • dfw.adaptive_end = 270,000 connections

• Once the start threshold is exceeded, existing flows are aged out faster.

• Once the end threshold is exceeded, flows are set to expire immediately to reduce the flow count beneath the end threshold. 

• You can issue the "vsipioctl gettimeout" command from the ESXi host that the Scanner VM resides on to see the  dfw.adaptive_start and dfw.adaptive_end values on a VM:
  
  *See Admin Guide Documentation for how to obtain the dvfilter to include in the below command

/bin/vsipioctl gettimeout -f nic-#######-eth0-vmware-sfw.2

Connection Timeouts:
dfw.tcp.first_packet : 120
dfw.tcp.opening : 30
dfw.tcp.established : 43200
dfw.tcp.closing : 120
dfw.tcp.fin_wait : 45
dfw.tcp.closed : 20
dfw.udp.first_packet : 60
dfw.udp.single : 30
dfw.udp.multiple : 60
dfw.icmp.first_packet : 20
dfw.icmp.error_reply : 10
dfw.other.first_packet: 60
dfw.other.single : 30
dfw.other.multiple : 60
dfw.ip.frag : 30
dfw.interval : 10
dfw.adaptive_start : 200000 <<<----
dfw.adaptive_end : 270000 <<<----
dfw.src_node : 0
dfw.ts_diff : 30
dfw.invalid_state : 600
dfw.invalid_l7 : 600
dfw.TBR_revalidation : 120

Resolution

• Option 1 (Recommended): Place the scanner VM in the DFW exclusion list.

• Option 2: Adjust dfw.adaptive_start and dfw.adaptive_end values on the scanner VM’s dvfilter.

  • Contact Broadcom Support for assistance with this change.

  • Note: Each host can handle a maximum of 2 million flow table entries. Increasing thresholds may cause the scanner to consume a disproportionate share of flow table resources, affecting other workloads.