NSX-T DFW does not process Challenge ACK correctly

Products

• VMware NSX

Issue/Introduction

• NSX-T Data Center 3.0.3, 3.1.1, 3.1.2 and 3.1.2.1
• TCP Handshake fails to complete when VM has DFW applied
  • Server VM with DFW applied receives a SYN packet for a port combination already tracked by an ESTABLISHED session
  • Server VM replies with a Challenge ACK
  • The Challenge ACK is dropped by the DFW and the client retransmits the SYN
• The DFW filter stats show the drop counters are increasing due to a sequence number validation issue

 1) Identify the slot 2 DFW filter of the VM using the command
  #summarize-dvfilter

  2) Check the stats on the VM's DFW filter e.g.
  #vsipioctl getfilterstat -f nic-123456-eth0-vmware-sfw.2
  PACKETS                       IN                OUT
  -------                       --                ---
  v4 pass:                 1330465278         1266757546
  v4 drop:                          3              97382  <<<<<<<
  v4 ackonsyn:                  75937                  0

  v6 pass:                          0                  0
  v6 drop:                          0                  0

  BYTES                         IN                OUT
  -----                         --                ---
  v4 pass:               451179707123       352330492310
  v4 drop:                        180            5334704
  v4 ackonsyn:                4555520                  0

  v6 pass:                          0                  0
  v6 drop:                          0                  0


  DROP REASON
  -----------
  state-mismatch:       97376
  src-limit:            42
  pkts-frag-queued-v4:  9
  seqno outside window: 2
  seqno old retrans:    97371 <<<<<<
  seqno gt maxack:      2
  seqno lt minack:      1

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

This behaviour is seen in a scenario where the server VM still tracks a TCP session that the client is no longer aware of.
When the server receives a SYN using ports already in use, it replies with a Challenge ACK.
The DFW drops the Challenge ACK. The client is unaware and retransmits the SYN packets until timeout.
This issue was introduced on NSX-T Data Center 3.0.3 and 3.1.1 and does not affect earlier releases on either branch.
On fixed versions, the DFW sees the Challenge ACK and sends a RST back to the server to clear the TCP state on the server.

Resolution

This issue is resolved in NSX-T Data Center 3.1.3.5, see Download Broadcom products and software.

Workaround:
Add the VM to the DFW exclusion list.