Receive packet drops (rx_no_bufs) on Cisco VIC Ethernet NIC (nenic) in VMware ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

This article addresses issues where VMware ESXi running on Cisco UCS servers experiences receive (RX) packet drops on physical vmnic adapters. These drops can often lead to Windows Failover Cluster heartbeat failures and service disruptions.

• Virtual machines, particularly Windows Failover Clusters, experience intermittent network connectivity loss or service failovers.
• Windows Event Logs report cluster heartbeat timeouts (UDP 3343).
• The esxcli network nic stats get -n vmnicX command shows high counts for Receive packets dropped.
• Host logs (hostd.log) contain warnings similar to:
  Wa(164) Hostd[2099472]: --> droppedRx: 1712724
• NIC private statistics command run on the ESX host show rx_no_bufs incrementing:
  vsish -e get /net/pNics/vmnicX/stats | grep rx_no_bufs rx_no_bufs

Environment

• VMware ESXi 
• Cisco UCS Servers
• Cisco VIC Ethernet NIC (nenic driver)

Cause

The rx_no_bufs statistic increments when the physical network interface (vmnic) exhausts its available receive ring buffers. This condition occurs when incoming packet bursts arrive faster than the ESXi kernel and network driver can process and dequeue them, leading to active packet drops at the ingress layer. The Windows Failover Clusters's use UDP for heartbeats, which lacks a retry mechanism, therefore, these drops lead to immediate cluster failovers.

Resolution

• The recommended solution is to increase the RX queues, the RX ring size, or a combination of both within the vNIC adapter policy in Cisco UCS Manager.
• Consult the Cisco vendor for assistance in configuring the appropriate RX and TX values. For more details, refer to the Cisco documentation: Cisco UCS VIC 15000 Series Best Practices in Ethernet Fabric White Paper

The ESXi host needs to be rebooted for these changes to take effect after updating the Cisco configuration. After increasing the Rx queues in the Cisco UCS Manager, the following command can be run to validate whether the values are reflected on the ESXi host. 

• For example, if the Rx queue is increased to 8, the rxqueue details will be shown as follows on the ESXi host.
  • [root@esx-01:~] vsish -e get /net/pNics/vmnic1/rxqueues/info
    rx queues info {
       # queues supported:8 >>>> These many Rx queues are available 
       # rss engines supported:0  >>>>>  This indicates device RSS is enabled.
       # filters supported:0
       # active filters:0
       # filters moved by load balancer:0
       RX filter classes: 0 -> No matching defined enum value found.
       Rx Queue features: 0 -> NONE
       Support restart queues:0
    }
• Run the following command on the ESXi host to list the RX queues; it should show 0-7 queues.
  • [root@esx-01:~] vsish -e ls /net/pNics/vmnic0/rxqueues/queues/
  • 0/
    1/
    2/
    3/
    4/
    5/
    6/
    7/

Additional Information

Tuning Guidelines for Cisco UCS Virtual Interface Cards