Multiple ESXi hosts dropping many packets - Rx packet drops - Cisco UCS servers

Products

• VMware vSphere ESXi

Issue/Introduction

- Host Servers are Cisco UCS servers

- You will be seeing many Rx drops on many of the ESXi hosts

- Logs on ESXi host for the uplinks shows Rx Drops and Out of buffers: (esxcli network nic stats get -n vmnic<X>)

NIC:  vmnicX
   vmnicX 0000:35:00.0 nenic Up Up 25000 Full 00:25:b5:##:##:## 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

    NIC Private statistics:

      tx_frames_ok: 18068021503
      tx_unicast_frames_ok: 18067380955
      tx_multicast_frames_ok: 111850
      tx_broadcast_frames_ok: 528698
      tx_bytes_ok: 12116707349919
      tx_unicast_bytes_ok: 12116660439455
      tx_multicast_bytes_ok: 15188584
      tx_broadcast_bytes_ok: 31721880
      tx_drops: 0
      tx_errors: 0
      tx_tso: 45910484
      rx_frames_ok: 17956912923
      rx_frames_total: 17958289962
      rx_unicast_frames_ok: 17927060390
      rx_multicast_frames_ok: 10854577
      rx_broadcast_frames_ok: 20374995
      rx_bytes_ok: 14213275034082
      rx_unicast_bytes_ok: 14213367289979
      rx_multicast_bytes_ok: 719449309
      rx_broadcast_bytes_ok: 1321584675
      rx_drop: 0 
      rx_no_bufs: 1377039        ----------------------> Out of buffers
      rx_errors: 0
      rx_rss: 0
      rx_crc_errors: 0
      rx_frames_64: 112321320
      rx_frames_127: 4360361450
      rx_frames_255: 3872426090
      rx_frames_511: 1184181826
      rx_frames_1023: 529054334
      rx_frames_1518: 811759454
      rx_frames_to_max: 2793218192
      tx_queue_[0]_frames_ok: 18037676191
      rx_queue_[0]_frames_ok: 17956912923
 

 Current Ring Size:
   RingInfo:
      RX: 512     ----> Current ring size
      RX Mini: 0
      RX Jumbo: 0
      TX: 256

- Same results for the other active uplink interface and also for the other ESXi hosts

- Tx and Rx queues are set to 8 and RSS is enabled already

Environment

VMware NSX

VMware vSphere 8.0U3

Cause

The Rx traffic is dropping due to small ring size on the uplink vmnic interfaces, causing out of buffers and in turn casing the drops on the interface

Resolution

Workaround:

- As these are UCS servers, per Cisco, the recommendation is to increase the ring size to be 4096 which allows for more packet buffering per queue, thus preventing dropped packets and retransmissions. Here is the reference from Cisco: https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/unified-computing-system-adapters/white-paper-c11-744754.html

Additional Information

VMNIC link may drop on Cisco UCS Host running certain nenic driver versions