Rx buffer exceeded on NICs

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machines are experiencing high network latency, poor network performance, or intermittent packet loss
  
  
• The esxcli network nic stats get -n vmnic# command shows rx_buffer_passed_threshold exceeded multiple times:
  
  NIC statistics for vmnic0:
      Packets received: 90397440
      Packets sent: 10477630
      Bytes received: 14683389816
      Bytes sent: 4400929475
      Receive packets dropped: 0
      Transmit packets dropped: 0
      Multicast packets received: 7923708
      Broadcast packets received: 64479979
      Multicast packets sent: 23613
      Broadcast packets sent: 851
      Total receive errors: 0
      Receive length errors: 0
      Receive over errors: 0
      Receive CRC errors: 0
      Receive frame errors: 0
      Receive FIFO errors: 0
      Receive missed errors: 0
      Total transmit errors: 0
      Transmit aborted errors: 0
      Transmit carrier errors: 0
      Transmit FIFO errors: 0
      Transmit heartbeat errors: 0
      Transmit window errors: 0

     NIC Private statistics:
           rx_buffer_passed_threshold: 417

Note: The output listed above contains Private statistics from Broadcom NICs and the output/headers might differ from other NICs.

Environment

ESXi 7.x
ESXi 8.x

Cause

A ring buffer is used to allocate a section of memory which is like a temporary holding area for packets whose packet rate may be so high that the code required to process them has trouble keeping up. The issue can occur when the Receive (Rx) ring buffer size for the physical vmnic is not large enough to handle the volume of the incoming network traffic. When the buffer is full, new packets are dropped, forcing retransmissions. 

Resolution

A ring buffer is used to allocate a section of memory which is like a temporary holding area for packets whose packet rate may be so high that the code required to process them has trouble keeping up. 

Each type of network adapter has a preset maximum which is determined by the device driver.  This is revealed by the above command featuring preset.

The preset maximum is usually higher than the default setting which is allocated when ESXi is installed.  This is revealed by the above command featuring current.

Below is an example of the output from the buffer commands above (the output may differ between vendors):

   Current Ring Size:
   RingInfo:
      RX: 1023
      RX Mini: 0
      RX Jumbo: 0
      TX: 1023

   Preset Maximum Ring Size:
   RingInfo:
      Max RX: 4095
      Max RX Mini: 0
      Max RX Jumbo: 0
      Max TX: 4095

In case the Rx buffer is being overwhelmed, increasing the physical uplink ring buffer sizes to the preset maximums will reduce the possibility of drivers being unable to cope with high packet arrival rates, however confirmation from the NIC vendor is recommended prior to making any changes.  

Command to increase both rx and tx:

esxcli network nic ring current set -n vmnic4 -r 4096 -t 4096

For receive ring buffer only:

esxcli network nic ring current set -n vmnic4 -r 4096

Max value will depend on the one got from the get command:

esxcli network nic ring preset get -n vmnicX

NOTE: Confirm with hardware vendor the recommended value to be set.