ESXi physical NIC Receive packets dropped due to rx discards at queue level
search cancel

ESXi physical NIC Receive packets dropped due to rx discards at queue level

book

Article ID: 436088

calendar_today

Updated On:

Products

VMware vSphere ESXi VMware vSphere ESX 8.x

Issue/Introduction

  • ESXi hosts experience a continuous increment in the "Receive packets dropped" counter for physical network adapters (vmnics).

  • This often manifests along with application slowness, RDP session drops, or SQL latency within virtual machines.

  • Advanced driver statistics reveal that these drops correlate specifically with the "discards rx" counter across multiple receive queues (e.g., rxq-rss or rxq-drss).  Example output from esxcli network nic stats get -n vmnicX:

    NIC statistics for vmnic0:
       Receive packets dropped: 11001
       [rxq-drss0] discards rx: 149
       [rxq-rss1] discards rx: 2508
       ...

    • The total "Receive packets dropped" is the aggregate sum of the "discards rx" counters from all individual hardware queues.

 

Environment

  • VMware ESXi 7.x, 8.x, and 9.x
  • VMware vSAN 7.x, 8.x, and 9.x

Cause

  • These discards occur when the ESXi networking stack cannot consume packets from the physical NIC fast enough to replenish the receive buffers.

  • This is typically not a driver defect but a result of one of the following:

    • High CPU Utilization: The ESXi host CPU load is too high, preventing the netpoll world from obtaining sufficient CPU cycles to poll the network queues and evacuate the buffers.

    • Network Microbursts: Sudden bursts of traffic fill the hardware receive queues faster than the system can process them, leading to buffer exhaustion.

Resolution

  1. Mitigate CPU Contention:

    • If drops are caused by CPU exhaustion, reduce the host load by migrating virtual machines to other hosts using vMotion or adding additional physical CPU resources. 

    • In environments experiencing severe ESXi CPU contention, evidenced by elevated CPU %RDY (Ready Time) metrics on virtual machines with low active CPU utilization, it is recommended to right-size the workloads by reducing their allocated vCPU count. This architectural optimization decreases the latency the virtual machine incurs while waiting for physical CPU (pCPU) allocation.

  2. Enable Flow Control (Pause Frames):

    • For bursty traffic, enable pause frames to allow the NIC to signal the switch to slow down transmission during congestion, reducing drops.

      esxcli network nic pauseParams set --auto 1 --rx 1 --tx 1 -n vmnicX
      • Note: This must also be configured and supported on the physical switch ports.

        • Increase the ring buffer size on the VM VMXnet3 adapter.

    • For bursty traffic of specific VM, increase the ring buffer size on the VMXnet3 adapter.

    • You can verify if a VM is "running out of buffers" by logging into the ESXi host via SSH and using the vsish (VMware Shell) utility.

      vsish -e get /net/portsets/[PortSet]/ports/[PortID]/vmxnet3/rxSummary | grep "running out of buffers"

  3. Increase RX Ring Buffers:

    • For bursty traffic, another way is to increase the RX ring buffer size to provide a larger cushion for bursts.

      esxcli network nic ring current set -n vmnicX -r 4095

  4. Inspect Physical Layer and Environmental Factors:

    If increasing the ring buffer size and mitigating CPU contention do not resolve the packet drops, investigate the physical network path:

    1. Validate Physical Hardware: Inspect and test the physical network cables and switchports connected to the affected vmnics. Faulty or degraded cabling can result in intermittent signal loss and packet discards that are reported at the driver level.

    2. Verify Switch Configuration: Ensure the physical switchport settings (speed, duplex, and MTU) match the ESXi host configuration.
    3. Perform Component Replacement: If drops persist on specific interfaces (e.g., vmnic6 or vmnic9), perform a test replacement of the network cable and migrate the connection to a known-good switchport to rule out environmental faults.

 

Additional Information

  • As long as microbursts exist on the physical network, these types of drops can be difficult to eliminate entirely. Ethernet is not a lossless medium by design; upper-layer protocols such as TCP include retransmission mechanisms to handle these occasional packet losses without data corruption.

Reference KB's:
ESXTOP overview for Performance Troubleshooting
Large packet loss (dropped packets in Virtual Machines) in the guest OS using VMXNET3 in ESXi

 

 

Powered by Wolken Software