Large number of "Receive packets dropped" & "Transmit packets dropped" on ESXi NIC's

Products

VMware vSphere ESXi

Issue/Introduction

Virtual machines are experiencing high network latency, poor network performance, or intermittent packet loss

The esxcli network nic stats get -n vmnic# command shows packet drops:

NIC statistics for vmnic0:

      Packets received: 90397440
      Packets sent: 10477630
      Bytes received: 14683389816
      Bytes sent: 4400929475
      Receive packets dropped: 44009 <<<<--------------
      Transmit packets dropped: 654643
      Multicast packets received: 7923708
      Broadcast packets received: 64479979
      Multicast packets sent: 23613
      Broadcast packets sent: 851
      Total receive errors: 0
      Receive length errors: 0
      Receive over errors: 0
      Receive CRC errors: 0
      Receive frame errors: 0
      Receive FIFO errors: 0
      Receive missed errors: 0
      Total transmit errors: 0
      Transmit aborted errors: 0
      Transmit carrier errors: 0
      Transmit FIFO errors: 0
      Transmit heartbeat errors: 0
      Transmit window errors: 0

Running the localcli --plugin-dir /usr/lib/vmware/esxcli/int networkinternal nic privstats get -n vmnicX command, returns counts for the rx_buffer_passed_threshold counter:

     NIC Private statistics:
           rx_buffer_passed_threshold: 417

- Note: The output listed above contains Private statistics from Broadcom NICs and the output/headers name might differ from other NICs.

Environment

VMware vSphere ESXi 8.x

Cause

The Receive missed errors (also known as Receive Packets Dropped) in an ESXi nicinfo report is a clear indicator of a performance bottleneck at the hardware or driver layer.

Note: A Ring Buffer is a fixed-size, circular data structure allocated in memory that serves as a high-speed staging area for incoming network frames. It acts as a critical buffer between the physical network interface (pNIC) and the ESXi kernel's network stack.

In high-throughput environments, the arrival rate of packets can occasionally exceed the rate at which the CPU can process them and clear the buffer. When the Receive (RX) ring buffer for a physical vmnic reaches its maximum capacity (total descriptors used), the NIC is unable to store subsequent frames. These frames are discarded at the hardware layer, a condition recorded as "Receive Missed Errors" or "RX Overruns."

Resolution

Each type of network adapter has a preset maximum which is determined by the device driver.

To identify the 'Current Ring Size':
esxcli network nic ring current get -n vmnicX

where 'X' is the network adapter ie vmnic0,1,2,3 etc

Example output:

Current Ring Size:
RingInfo:
RX: 1023
RX Mini: 0
RX Jumbo: 0
TX: 1023

To identify the "Preset Maximum Ring Size":
esxcli network nic ring preset get -n vmnicX

where 'X' is the network adapter ie vmnic1,2,3 etc

Example output:

Preset Maximum Ring Size:
RingInfo:
Max RX: 4095
Max RX Mini: 0
Max RX Jumbo: 0
Max TX: 4095

Note: The Preset Maximum defines the hardware-enforced upper limit for the RX buffer configuration; the actual buffer size cannot exceed this threshold. Additionally, the value may differ between vendors.

In case the Rx buffer is being overwhelmed, increasing the physical uplink ring buffer sizes to the preset maximums will reduce the possibility of drivers being unable to cope with high packet arrival rates, however confirmation from the NIC vendor is recommended prior to making any changes.

To modify the ring buffer settings on an ESXi host, use the following esxcli commands.

Note: Before applying these changes, identify the hardware limits by running the "preset get" command.

Verify Hardware Limits:
esxcli network nic ring preset get -n vmnicX

Update Configuration:

To increase both RX and TX buffers simultaneously: esxcli network nic ring current set -n vmnicX -r 4096 -t 4096
To adjust the Receive (RX) buffer only: esxcli network nic ring current set -n vmnicX -r 4096
To adjust the Transmit (TX) buffer only: esxcli network nic ring current set -n vmnicX -t 4096

Replace vmnicX with the target identifier and ensure the value (e.g., 4096) does not exceed the "Preset Maximum" retrieved in the verification step.

Note: The value of 4096 used in the command above is an example based on a typical 'Preset Maximum Ring Size.' It is critical to verify the specific maximum supported by your hardware and consult with your hardware vendor for recommended configurations before applying changes.