Mellanox ConnectX-4: Resolving vSAN High pNIC Error Rate Alarms (nmlx5_core)

Products

• VMware vSAN

Issue/Introduction

An ESXi 8.0 Update 3 host using Mellanox ConnectX-4 Lx adapters may trigger a vSAN health alarm indicating high pNIC error rates. This occurs when the physical NIC hardware reports buffer overflows that the management layer interprets as active errors, even if no performance impact is currently felt.

Symptoms:

• Alarm: "High pNic error rate detected. Check the host's vSAN performance view for details."
• The command esxcli network nic stats get -n vmnicX shows incrementing Receive missed errors.
• NIC private statistics show the outOfBuffer counter matching the missed errors exactly.

Environment

• VMware vSphere ESXi 8.0 Update 3
• VMware vSAN 8.x
• Mellanox ConnectX-4 Lx EN (nmlx5_core)
• Firmware: Versions prior to 14.32.1900

Cause

The "Receive missed errors" are caused by an outOfBuffer condition where the NIC hardware lacks sufficient buffer space to process incoming packets during traffic bursts. On firmware versions prior to 14.32.1900, these hardware-level drops are not always correctly synchronized with the ESXi driver, leading to cumulative alarms in vCenter.

Resolution

Fixed in Mellanox firmware version 14.32.1900 and higher. 

1. Verify the supported driver and firmware combination for your specific hardware on the Broadcom Compatibility Guide.
2. Download the required firmware directly from your hardware vendor's support site (e.g., Dell, HPE, or NVIDIA/Mellanox).
3. Reboot the host to clear the current hardware error counters.
4. Check the current ring buffer settings:
   
   

   esxcli network nic ring current get -n vmnicX

5. If errors persist, increase the RX ring buffer gradually. It is recommended to test in increments (e.g., 2048 or 4096) rather than jumping immediately to 8192 to avoid excessive memory consumption:
   
   

   esxcli network nic ring current set -r 4096 -n vmnicX

   
   
6. Monitor host memory usage and the vSAN performance view to confirm the alarm remains cleared.

Additional Information

Related Articles: 

• Troubleshooting NIC errors
• Receive Missed Errors on Mellanox adapters

If errors persist, see Contact Broadcom Support.