Packet drops observed on VMXNET3 vNIC due to Distributed Firewall policy - VMware VCF

Products

• VMware vSphere ESXi
• VMware NSX

Issue/Introduction

• Standard network troubleshooting shows packet loss at the virtual interface (vNIC) layer
  
  
• ESXi `esxtop` command shows non-zero values in the `%DRPRX` (dropped receive) or `%DRPTX` (dropped transmit) columns for specific VM worlds

• There are no packet drops noticed at the vmnic level.

• There are no ring-full or out-of-buffer errors seen on the VMs.

• There are real time packet drops observed as per the performance charts in vCenter
  
  

 

Environment

• VMware vSphere ESXi 8.x
• VMware NSX-T / NSX 4.x

Cause

Packet drops at the virtual port level are often the result of explicit security policies configured within the NSX Distributed Firewall (DFW). When network traffic does not match an 'Allow' rule, or explicitly matches a 'Drop' or 'Reject' rule, the `vmware-sfw` filter discards the frames at the I/O chain boundary before they reach the Guest OS (for Rx) or the physical network (for Tx).

Resolution

To verify if packet drops are caused by firewall policy, follow these steps:

1. Identify the VM Port ID by running the following command on the ESXi host:

   net-stats -l

2. Retrieve VSI statistics for the port to identify where the drop occurs:

   vsish -e get /net/portsets/DvsPortset-#/ports/<PortNum>/vmxnet3/rxSummary

   Note: Replace DvsPortset-# and <PortNum> with the values found in step 1.
3. Identify the specific DFW filter name for the VM vNIC:

   summarize-dvfilter | grep -A2 <VM_Name>

4. Query the filter statistics to confirm policy-based discards (Matches):

   vsipioctl getfilterstat -f nic-########-eth0-vmware-sfw.2

   Look for non-zero values in the v4 drop or v4 reject rows.
5. Compare the total Rx drop count against the vsipioctl getfilterstats output count. If these numbers are nearly identical, the drop is occurring within the firewall filter.
   
   Note: Distributed Firewall drop counts are migrated with the VM during a vMotion; therefore, the reported drops may not have occurred exclusively on the current host. 
   
   
6. In the NSX Manager UI, navigate to Security > Distributed Firewall and review the rules applied to the VM.
7. Identify the rule causing the discard.
8. If the drops are unintended, modify the security policy to Allow the affected traffic.

If the drops are confirmed to be intentional by policy, no further action is required.

Additional Information

If assistance is needed to identify a specific rule, see Contact Broadcom support

To be updated on changes to this information, please subscribe to this article. For instructions, see Subscribe to a Broadcom knowledge article by article or product.