F5 Load Balancers and other 3rd party appliances are subject to drop packets intermittently at the DFW VSIP module within ESXI
Article ID: 383731
Updated On:
Products
Issue/Introduction
Packets drops are observed which result in re-transmissions between Server and Client when the distributed firewall in NSX is applied to F5/3rd party Load Balancers.
Environment
VMware NSX
Cause
A TCP stream will collapse due to re-transmissions on a given segment This is outlined in the trace below wherein this particular packet is lost after passing through DVFilterTxCompletionCB which exists as a function of the slot-2 filter in the I/O chain for the Distributed Firewall.
This issue manifests as TCP seq/ack num errors, which are usually caused by asymmetric traffic on conjunction with DFW packet processing. Relevant statistics for the slot-2 filter are as follows and can be found within the /commands/vsipioctl_info.sh.txt file in the ESXI host bundle.
The TCP seq/ack num errors for the F5 switchport in this case are as follows which backs up the trace capture output above:
/bin/vsipioctl getfilterstat -f nic-########-eth1-vmware-sfw.2
PACKETS IN OUT
------- -- ---
v4 pass: 80231107367 81168845167
v4 drop: 649138349 545285056
v4 reject: 2 28
v4 ackonsyn: 9 0
v6 pass: 3938667 125
v6 drop: 1 0
L2 drop: 0 0
BYTES IN OUT
----- -- ---
v4 pass: 107169461783642 107123376995907
v4 drop: 111265053213 107723847
v4 reject: 92 1572
v4 ackonsyn: 524 0
v6 pass: 545913199 10720
v6 drop: 152 0
L2 drop: 0 0
DROP REASON
-----------
fragment: 33
state-insert: 259
strict no syn: 27317673
icmp error: 157599
state-mismatch: 80047139
3wh error: 30
seqno outside window: 80044298
seqno old retrans: 2595
seqno old ack: 17
seqno bad ack: 14
seqno gt maxack: 80044296
seqno lt minack: 2500
Relevant statistics can also be found within Vsish on the ESXI host:
/net/portsets/DvsPortset-0/ports/########/> cat /net/portsets/DvsPortset-0/ports/########/inputStats
io chain stats {
starts:139002125
resumes:0
inserts:0
removes:0
errors:0
pktsStarted:282415348
pktsPassed:41274213
pktsDropped:57229
pktsCloned:0
pktsFiltered:241083906
pktsFaulted:0
pktsQueued:0
pktErrors:0
pktsInjected:0
functions:
DFW:
DVFILTER_VNIC_IN_GUEST <vmware-sfw:0x############>
pktsStarted:282415348
pktsPassed:282358183
pktsDropped:57165
pktsFiltered:0
pktsQueued:0
pktsFaulted:0
pktsInjected:0
pktErrors:0
pktsBypassed:0
Resolution
Place the F5 Load Balancer or other 3rd party Load balancers into the DFW exclusion list : https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/administration-guide/security/distributed-firewall/manage-a-firewall-exclusion-list.html
if E-W Service Insertion (SI) is being used, place the VMs in the SI exclude list as well : https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/administration-guide/security/east-west-network-security-chaining-third-party-services/exclude-members-from-a-security-service.html
Additional Information
Please refer to the following document wherein Broadcom's official recommendation is to place "Virtual machines such as load balancers, firewalls, virtual network functions (routing, switching, etc.), and any virtual machines that require promiscuous mode must be in a DFW Exclusion list." Manage a Firewall Exclusion List
Feedback
