Packet loss observed on North/South traffic between NSX Edge nodes on Edge Host

Products

• VMware NSX

Issue/Introduction

• Packet loss affecting North/South or South/North sessions through NSX Edge nodes 
• When analyzing traffic, packet loss is occurring between the ESXi Host where the Edge node resides and Edge node VM
• Missing packets are seen entering the ESXi Host uplink, but is not seen on the Edge node VMs
• Edge nodes were originally deployed from OVA and not from the NSX UI
• Distributed Firewall is being used within NSX
• Confirm DFW rules are being applied to the Edge node vNics:
  
  • Use the commands to check if there are rules being applied
    summarize-dvfilter | grep -A9 <Edge VM hostname>
vsipioctl getrules -f <slot 2 name from above command>
  • Check for all Edge node vNICs from the summarize-dvfilter command
  • If there is no slot 2 name or if the second command is blank then no rules are being applied

Environment

VMware NSX 4.x

Cause

Since Edge nodes were originally deployed from OVA and not from the NSX UI, they were not automatically added to the DFW Exclusion List. This causes the host to potentially check DFW rules for all traffic to and from the Edge nodes. This can cause high amount of DFW memory usage, despite a low amount of traffic overall. This can cause the host to drop packets when all allowable allocated DFW memory is utilized. 

Resolution

Add the Edge node VMs to the DFW Exclusion List:

• In the NSX UI, go to Security > Settings > Distributed Firewall > Exclusion list
• Create a group with all the Edge nodes
• Add the group to the Exclusion List

Additional Information

• Check for "Destroy session" messages within the vmkernel.log on the ESXi hosts where the Edge nodes reside:

In(182) vmkernel: cpu43:16929810)Net: 2621: Created session 14 successfully.
In(182) vmkernel: cpu43:16929810)Net: 3808: Filter tuple 5, 4
In(182) vmkernel: cpu43:16929810)Net: 3808: Filter tuple 6, 10.##.##.##
In(182) vmkernel: cpu43:16929810)Net: 3808: Filter tuple 8, 10.##.##.##
In(182) vmkernel: cpu3:2097296)Net: 2733: Destroy session 14 successfully.

• • This may indicate packets being dropped from the host
    
    
• Check filter stats on the same slot 2 name and check for a high amount drop memory count:
  
  summarize-dvfilter | grep -A9 <Edge VM hostname>
/bin/vsipioctl getfilterstat -f <slot 2 name from the above command>

 

• • Check for the following lines:

DROP REASON
-----------
memory:               27521