Alarm "High pNic error rate detected" on hosts in vSAN clusters using Mellanox NICs during vMotion Bursts

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi hosts report "High pNic error rate detected" 
• Analyzing NIC statistics using esxcli network nic stats get -n vmnicX shows an incrementing "Receive missed errors" counter.
• Detailed private statistics via /usr/lib/vmware/vm-support/bin/nicinfo.sh show an incrementing outOfBuffer counter.
• Performance issues or alarms coincide with high-concurrency vMotion activities.

Environment

VMware vSphere ESXi

Cause

Short-term burst network traffic during high-concurrency vMotion operations can exhaust the hardware RX ring buffer of Mellanox ConnectX-5 adapters. Since ESXi 8.0 Update 3, updated nmlx5_core drivers correctly reflect these "out of buffer" events as "Receive Missed Errors," which triggers the vCenter alarm.

Resolution

1. Reduce vMotion Concurrency: Adjust the vMotion settings to "Less Frequent vMotions" to reduce simultaneous traffic concurrency.
2. Optimize the settings for pNic and Physical switches if necessary: https://knowledge.broadcom.com/external/article/392333

Additional Information

https://knowledge.broadcom.com/external/article/393164