DFW is dropping packets by unexpected ack between SYN and SYN/ACK when TCP window scaling option is enabled

Products

• VMware NSX

Issue/Introduction

Symptoms:

• DFW drops packets
• The server responds ACK before SYN/ACK in TCP 3 way handshake
• TCP Windows Scaling is enabled
• The counter "seqno outside window"in filter stats is incremented

 

TCP 3 way handshake where ACK comes before SYN/ACK looks following:

192.168.0.11 -> 192.168.0.12 TCP 74 [TCP Retransmission] 19222 > http [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=1841494880 TSecr=0 WS=128

192.168.0.12 -> 192.168.0.11 TCP 66 [TCP Dup ACK 118613#1] http > 19222 [ACK] Seq=1469843982 Ack=1 Win=49232 Len=0 TSval=3042767310 TSecr=1841492876

192.168.0.12 -> 192.168.0.11 TCP 78 [TCP Retransmission] http > 19222 [SYN, ACK] Seq=1469843981 Ack=1 Win=49232 Len=0 TSval=3042767449 TSecr=1841492876 MSS=1460 WS=1 SACK_PERM=1

192.168.0.11 -> 192.168.0.12 TCP 66 [TCP ACKed unseen segment] 19222 > http [ACK] Seq=1 Ack=1469843982 Win=29312 Len=0 TSval=1841496261 TSecr=3042767449

 

To display filter stats, use vsipioctl on ESXi CLI

# vsipioctl getfilterstat -f <filter name>

DROP REASON

-----------

state-mismatch:      8

seqno outside window: 8

Environment

VMware NSX-T Data Center

Cause

DFW fails to recognize correct TCP window calculated from scaling factors by unexpected ACK in middle of 3 way handshake

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.3.1 and VMware NSX 4.1.1.