ESXi physical NIC Receive packets dropped due to rx discards at queue level

Products

VMware vSphere ESXi

Issue/Introduction

ESXi hosts experience a continuous increment in the "Receive packets dropped" counter for physical network adapters (vmnics). Advanced driver statistics reveal that these drops correlate specifically with the "discards rx" counter across multiple receive queues (e.g., rxq-rss or rxq-drss).

Example output from esxcli network nic stats get -n vmnicX:

NIC statistics for vmnic0:
Receive packets dropped: 11001
[rxq-drss0] discards rx: 149
[rxq-rss1] discards rx: 2508
...

The total "Receive packets dropped" is the aggregate sum of the "discards rx" counters from all individual hardware queues.

Environment

VMware vSphere ESXi

Cause

These discards occur when the ESXi networking stack cannot consume packets from the physical NIC fast enough to replenish the receive buffers. This is typically not a driver defect but a result of one of the following:

High CPU Utilization: The ESXi host CPU load is too high, preventing the netpoll world from obtaining sufficient CPU cycles to poll the network queues and evacuate the buffers.
Network Microbursts: Sudden bursts of traffic fill the hardware receive queues faster than the system can process them, leading to buffer exhaustion.

Resolution

Mitigate CPU Contention:

If drops are caused by CPU exhaustion, reduce the host load by migrating virtual machines to other hosts using vMotion or adding additional physical CPU resources.

In environments experiencing severe ESXi CPU contention—evidenced by elevated CPU %RDY (Ready Time) metrics on virtual machines with low active CPU utilization—it is recommended to right-size the workloads by reducing their allocated vCPU count. This architectural optimization decreases the latency the virtual machine incurs while waiting for physical CPU (pCPU) allocation.

Enable Flow Control (Pause Frames):

For bursty traffic, enable pause frames to allow the NIC to signal the switch to slow down transmission during congestion, reducing drops.

esxcli network nic pauseParams set --auto 1 --rx 1 --tx 1 -n vmnicX

Note: This must also be configured and supported on the physical switch ports.

Increase the ring buffer size on the VM VMXnet3 adapter.

For bursty traffic of specific VM, increase the ring buffer size on the VMXnet3 adapter.

Refer the following KB: Large packet loss (dropped packets in VM) in the guest OS using VMXNET3 in ESXi

You can verify if a VM is "running out of buffers" by logging into the ESXi host via SSH and using the vsish (VMware Shell) utility.

vsish -e get /net/portsets/[PortSet]/ports/[PortID]/vmxnet3/rxSummary | grep "running out of buffers"

Increase RX Ring Buffers:

For bursty traffic, another way is to increase the RX ring buffer size to provide a larger cushion for bursts.

esxcli network nic ring current set -n vmnicX -r 4095

Additional Information

As long as microbursts exist on the physical network, these types of drops can be difficult to eliminate entirely. Ethernet is not a lossless medium by design; upper-layer protocols such as TCP include retransmission mechanisms to handle these occasional packet losses without data corruption.

Reference KB:

https://knowledge.broadcom.com/external/article/308290/esxtop-overview-for-performance-troubles.html

https://knowledge.broadcom.com/external/article/324556/