ESXi Host is seen with high number of Receive Packets Dropped on Specific Physical NIC

Products

• VMware NSX

Issue/Introduction

A large number of "Receive packets dropped" are occurring on an ESXi host. The issue is environment-isolated to a single host and causes an active alert within VMware Aria Operations for Networks indicating critical packet loss on the physical boundary.

Analysis of the host network statistics confirms that the dropped packets are strictly incrementing on a specific host. The esxcli network nic stats get command shows high drop counts:

[root@host:~] esxcli network nic stats get -n vmnic2
NIC statistics for vmnic2
   Packets received: 88924040750
   Packets sent: 69870283494
   Bytes received: 107018966907299
   Bytes sent: 74345629163764
   Receive packets dropped: 681455735  <<<<<<------------
   Transmit packets dropped: 0
   Multicast packets received: 654484574
   Broadcast packets received: 494334834
   Multicast packets sent: 62009
   Broadcast packets sent: 32014

Reviewing deeper driver statistics highlights a non-zero value for the threshold counter

vsish -e get /net/portsets/ENK10_CLIENT_VDS01/ports/67108897/vmxnet3/rxSummary

:
:
rx_bits: 204035916044800
rx_buffer_passed_threshold: 417    <<<<<<-------------
rx_pcs_symbol_err: 0
:
:

Environment

VMware NSX-T Data Center
VMware NSX

Cause

The physical network interface card (NIC) is receiving traffic faster than the ESXi kernel driver can clear the rx descriptors, leading to hardware FIFO buffer exhaustion indicated by an incrementing rx_buffer_passed_threshold.

Resolution

 

• Identify the virtual machines residing on the impacted ESXi host generating the highest packets-per-second (PPS) or network throughput.

• Execute a live vMotion migration to move lower-priority virtual machines off the affected ESXi host to alternate nodes within the cluster. This decreases the aggregate ingress packet arrival rate on the host.

• Monitor the network interface statistics to confirm if the drop rate stabilizes or stops entirely:

  esxcli network nic stats get -n <vmnic_name>
  

• Also, check the Hardware Compatibility List (HCL) to ensure the network card's driver and firmware versions are fully aligned and supported.