High Packet Loss Observed in Aria Ops for Networks and vSphere Advanced Metrics when NSX IDS/IPS is Enabled

Products

• VMware NSX
• VMware vDefend Firewall

Issue/Introduction

High packet loss counts are observed when NSX IDS/IPS is enabled. Specifically, the issue arises because the calculation of dropped packets incorrectly excludes the pktsFaulted counter, which represents the number of packets redirected to the IDPS engine. This miscalculation results in a higher-than-accurate packet loss count.

To confirm the issue, issue the following command:
 
vsish -e get /net/portsets/<portset>/ports/<port#>/inputStats

 

The 'portset' and 'port#' variables can be retrieved by issuing the 'net-stats -l' on the ESXi host. 

 

Example:

vsish -e get /net/portsets/DvsPortset-2/ports/134217760/inputStats
io chain stats {
   starts:1691416
   resumes:1157194
   inserts:0
   removes:0
   errors:0
   pktsStarted:1921739
   pktsPassed:84832
   pktsDropped:1824904   <------- Should be 154    
   pktsCloned:0
   pktsFiltered:1836753
   pktsFaulted:1824750    
   pktsQueued:0
   pktErrors:0
   pktsInjected:1824750
   
   
 
The correct calculation is:

pktsDropped = pktsStarted + pktsInjected - pktsPassed - pktsFiltered - pktErrors - pktsFaulted

The incorrect calculation is: 

pktsDropped = pktsStarted + pktsInjected - pktsPassed - pktsFiltered - pktErrors

So pktsDropped should = 154

But from the above we see pktsDropped = 1824904

Environment

VMware NSX w/IDPS enabled (any version)

ESXi 7.0 u3q - Build 23794027

 

Cause

Incorrect calculation of dropped packets. 

Resolution

Fix is currently targeted for:

ESXi 7.0 P10
ESXi 8.0 P05
ESXi 9.x