vCenter (VC) or vROps may incorrectly report small packet drops for a Distributed Virtual Switch (DVS) port, even when no actual packet drops occur.

Products

• VMware vSphere ESXi
• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• In vsish, you do not see dropped packets under vmxnet3 statistics, nor is there any increment in the "droppedRx" or "droppedTx" counter under overall port statistics.
• Virtual machines migrated to ESXi version 7.0 do not exhibit this false reporting.
• You may observe fluctuations in the pktsOutDropped and pktsInDropped counters, which temporarily increase to a non-zero value before returning to zero.
• In Aria Operations there might be regular alerts about high transmitted packet drops on VMs.
• In Aria Operations there might be TX packet drop spikes reported in the 'Network|Total Transmitted Packets Dropped' graph.
• In vCenter in the Monitor-> Performance-> Advanced-> Real-time-> Network-> Transmit packets dropped graph there might be spikes reported but not the expected constant non-zero line.

Environment

VMware ESXi 8.0
VMware ESXi 9.0
Aria Operations

Cause

• Essentially, the counters being reported to vCenter from the host are inaccurate which means Aria Ops has the same inaccurate data, since it pulls the data from vCenter.
• In some cases this can lead to alarms being reported in Aria Operations. The cause for the host reporting inaccurate counters is the following:
• As packets traverse through vNIC backend, and the network stack including the vswitch module,  there can be drops in any of the network modules. The packet drops are calculated by merging all drops from all the network modules. However as some of the modules does not track the drops directly, the drops are calculated by several counters which can not be read atomically. This can lead to false positive drops. In this context, atomically means "all at once" or as a single, synchronized snapshot.

Resolution

• While there is no immediate fix, customers who monitor the DVS port statistics graph should follow this guidance:
  • Ignore temporary spikes in the drop count.
  • A constant, non-zero line indicates actual packet drops.
• There are improvements for this issue in versions 8.0.3P08 and 9.0.2. It is recommended to update to a version with the improvements which will provide more accurate reporting of counters from the hosts.
• There is a redesign planned for how drops are counted in a future release which will address the root cause of this inaccurate reporting.