Multiple ESXi hosts dropping many packets - Rx packet drops - Cisco UCS servers
Article ID: 412616
Updated On:
Products
Issue/Introduction
- The ESXi host Servers are Cisco UCS servers
- "Receive Packet Dropped" counters are reported to be increasing for physical adapters across ESXi hosts.
- Logs on ESXi host for the uplinks shows "Receive Packet Dropped" and "Out of buffers counters as high.
- To check the "Receive Packet Dropped" on live ESXi host, please run the command:
esxcli network nic stats get -n vmnic<X>
NIC: vmnicX vmnicX 0000:35:00.0 nenic Up Up 25000 Full 00:25:b5:##:##:## 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
NIC Private statistics:
tx_frames_ok: 18068021503 tx_unicast_frames_ok: 18067380955 tx_multicast_frames_ok: 111850 tx_broadcast_frames_ok: 528698 tx_bytes_ok: 12116707349919 tx_unicast_bytes_ok: 12116660439455 tx_multicast_bytes_ok: 15188584 tx_broadcast_bytes_ok: 31721880 tx_drops: 0 tx_errors: 0 tx_tso: 45910484 rx_frames_ok: 17956912923 rx_frames_total: 17958289962 rx_unicast_frames_ok: 17927060390 rx_multicast_frames_ok: 10854577 rx_broadcast_frames_ok: 20374995 rx_bytes_ok: 14213275034082 rx_unicast_bytes_ok: 14213367289979 rx_multicast_bytes_ok: 719449309 rx_broadcast_bytes_ok: 1321584675 rx_drop: 0 rx_no_bufs: 1377039 ----------------------> Out of buffers rx_errors: 0 rx_rss: 0 rx_crc_errors: 0 rx_frames_64: 112321320 rx_frames_127: 4360361450 rx_frames_255: 3872426090 rx_frames_511: 1184181826 rx_frames_1023: 529054334 rx_frames_1518: 811759454 rx_frames_to_max: 2793218192 tx_queue_[0]_frames_ok: 18037676191 rx_queue_[0]_frames_ok: 17956912923
Current Ring Size: RingInfo: RX: 512 ----> Current ring size RX Mini: 0 RX Jumbo: 0 TX: 256
- Similarly other adapters on the same ESXi host as well as across different ESXi hosts also report the receive packet dropped counters as high.
- Tx and Rx queues are set to 8 and RSS is enabled already for the concerned adapters.
- The RSS queues for the adapters are being used successfully for incoming traffic, which can be validated by checking the stats of the "
rx_rss_queue_[x]_frames_ok]" for the vmnics. - The hostd log file in ESXi host shows the droppedRX counter increasing.
hostd.log:2025-MM-DDTHH:MM:SS.MMMZ Wa(AAA) Hostd[XXXXXXX]: [Originator@ZZZZ sub=Statssvc.StatsCollector] Error stats for pnic: vmnicN hostd.log:2025-MM-DDTHH:MM:SS.MMMZ Wa(AAA) Hostd[YYYYYYY]: --> droppedRx: 58879
Environment
VMware NSX
VMware vSphere 8.0U3
Cause
The Rx traffic is dropping due to small ring size on the uplink vmnic interfaces, causing out of buffers and in turn causing the drops on the interface
Resolution
Workaround:
As these are UCS servers, per Cisco, the recommendation is to increase the ring size to be 4096 which allows for more packet buffering per queue, thus preventing dropped packets and retransmissions.
Here is the reference from Cisco: https://www.cisco.com/c/en/us/products/collateral/interfaces-modules/unified-computing-system-adapters/white-paper-c11-744754.html
If packet drops continue even after maximizing adapter ring sizes, enabling RSS, and updating your drivers and firmware, the host is likely receiving more traffic than it can handle.
To resolve this, please balance the heavily utilized VMs across different ESXi hosts in the cluster. This prevents a single host from becoming saturated with incoming traffic, which fills the receive queues and causes packets to drop.
Additional Information
The commands "esxcli network nic list" shows the Network Interface Cards Information below for Cisco nenic network card.
Name PCI Device Driver Admin Status Link Status Speed Duplex MAC Address MTU Description
------ ------------ ------ ------------ ----------- ----- ------ ----------------- ---- -----------
vmnic0 0000:XX:XX.0 nenic Up Up 20000 Full XX:XX:XX:XX:XX:XX 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic1 0000:XX:XX.1 nenic Up Up 20000 Full YY:YY:YY:YY:YY:YY 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic2 0000:XX:XX.2 nenic Up Up 20000 Full ZZ:ZZ:ZZ:ZZ:ZZ:ZZ 9000 Cisco Systems Inc Cisco VIC Ethernet NICFeedback
