Load Balancers and other 3rd party appliances are subject to drop packets intermittently at the DFW VSIP module within ESXI

Products

• VMware NSX
• VMware vDefend Firewall
• VMware NSX-T Data Center

Issue/Introduction

• Packets drops are observed which result in re-transmissions between Server and Client when the distributed firewall in NSX is applied to F5 / AVI SE's / 3rd party Load Balancers. 

Environment

VMware NSX
VMware vDefend Firewall

Cause

A TCP stream will collapse due to re-transmissions on a given segment This is outlined in the trace below wherein this particular packet is lost after passing through DVFilterTxCompletionCB which exists as a function of the slot-2 filter in the I/O chain for the Distributed Firewall. 

This issue manifests as TCP seq/ack num errors, which are usually caused by asymmetric traffic on conjunction with DFW packet processing. Relevant statistics for the slot-2 filter are as follows and can be found within the /commands/vsipioctl_info.sh.txt file in the ESX host bundle. 

The TCP seq/ack num errors for the F5 switchport in this case are as follows which backs up the trace capture output above:

/bin/vsipioctl getfilterstat -f nic-########-eth1-vmware-sfw.2
PACKETS                       IN                OUT
-------                       --                ---
v4 pass:                80231107367        81168845167
v4 drop:                  649138349          545285056
v4 reject:                        2                 28
v4 ackonsyn:                      9                  0

v6 pass:                    3938667                125
v6 drop:                          1                  0

L2 drop:                          0                  0

BYTES                         IN                OUT
-----                         --                ---
v4 pass:            107169461783642    107123376995907
v4 drop:               111265053213          107723847
v4 reject:                       92               1572
v4 ackonsyn:                    524                  0

v6 pass:                  545913199              10720
v6 drop:                        152                  0

L2 drop:                          0                  0

DROP REASON
-----------
fragment:             33
state-insert:         259
strict no syn:        27317673
icmp error:           157599
state-mismatch:       80047139
  3wh error:            30
  seqno outside window: 80044298
  seqno old retrans:    2595
  seqno old ack:        17
  seqno bad ack:        14
  seqno gt maxack:      80044296
  seqno lt minack:      2500

Relevant statistics can also be found within Vsish on the ESX host:

/net/portsets/DvsPortset-0/ports/########/> cat /net/portsets/DvsPortset-0/ports/########/inputStats
io chain stats {
   starts:139002125
   resumes:0
   inserts:0
   removes:0
   errors:0
   pktsStarted:282415348
   pktsPassed:41274213
   pktsDropped:57229
   pktsCloned:0
   pktsFiltered:241083906
   pktsFaulted:0
   pktsQueued:0
   pktErrors:0
   pktsInjected:0
   functions:

DFW:

DVFILTER_VNIC_IN_GUEST <vmware-sfw:0x############>
                pktsStarted:282415348
                pktsPassed:282358183
                pktsDropped:57165
                pktsFiltered:0
                pktsQueued:0
                pktsFaulted:0
                pktsInjected:0
                pktErrors:0
                pktsBypassed:0

 

Resolution

Place the F5 Load Balancer or other 3rd party Load Balancers into the DFW exclusion list : https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/manage-a-firewall-exclusion-list.html

if E-W Service Insertion (SI) is being used, place the VMs in the SI exclude list as well : https://techdocs.broadcom.com/us/en/vmware-security-load-balancing/vdefend/vdefend-firewall/4-2/east-west-network-security-chaining-third-party-services/exclude-members-from-a-security-service.html

If the backend server sends the response to the client directly, then the backend servers need to be added to the DFW exclusion list as well.

Additional Information

Please refer to the following document wherein Broadcom's official recommendation is to place "Virtual machines such as load balancers, firewalls, virtual network functions (routing, switching, etc.), and any virtual machines that require promiscuous mode must be in a DFW Exclusion list." Manage a Firewall Exclusion List