High Packet Loss Observed in Aria Ops for Networks and vSphere Advanced Metrics when NSX IDS/IPS or L7 DFW is Enabled

Products

• VMware NSX
• VMware vDefend Firewall
• VCF Operations/Automation (formerly VMware Aria Suite)
• VMware Aria Operations (formerly vRealize Operations) 8.x

Issue/Introduction

When NSX IDS/IPS or L7 DFW is enabled, there may be a couple of symptoms:

1. Alert "VM Network has dropped transmit packets" is triggered in Aria Ops for Network:
   
2. High packet loss counts are observed in Aria Ops for Networks and vSphere Advanced Metrics

Environment

VMware NSX w/IDPS or L7 DFW is enabled (any version)

ESXi 7.0 versions prior to ESXi 7.0 P10

ESXi 8.0 versions prior to ESXi 8.0 P05

 

Cause

The issue arises because the calculation of dropped packets incorrectly excludes the pktsFaulted counter, which represents the number of packets redirected to the IDPS engine or L7 process in userspace. This miscalculation results in a higher-than-accurate packet loss count.

To confirm the issue, issue the following command:
 
vsish -e get /net/portsets/<portset>/ports/<port#>/inputStats

 

The 'portset' and 'port#' variables can be retrieved by issuing the 'net-stats -l' on the ESXi host. 

 

Example:

vsish -e get /net/portsets/DvsPortset-#/ports/#########/inputStats
io chain stats {
   starts:1691416
   resumes:1157194
   inserts:0
   removes:0
   errors:0
   pktsStarted:1921739
   pktsPassed:84832
   pktsDropped:1824904   <------- Should be 154    
   pktsCloned:0
   pktsFiltered:1836753
   pktsFaulted:1824750    
   pktsQueued:0
   pktErrors:0
   pktsInjected:1824750
   
   
 
The correct calculation is:

pktsDropped = pktsStarted + pktsInjected - pktsPassed - pktsFiltered - pktErrors - pktsFaulted

The incorrect calculation is: 

pktsDropped = pktsStarted + pktsInjected - pktsPassed - pktsFiltered - pktErrors

So pktsDropped should = 154

But from the above we see pktsDropped = 1824904

Resolution

Fix is included in the following versions and later:

ESXi 7.0 P10
ESXi 8.0 P05