Receive Missed Errors detected on Mellanox pNICs after updating the driver or ESXi
Article ID: 393164
Updated On:
Products
Issue/Introduction
- After upgrading Mellanox NIC drivers, you may see "High pNIC Error rate detected" within vCenter shortly after.
- Run the following command to inspect the NIC statistics:
#esxcli network nic stats get -n vmnic# - You may see numerous 'Receive Missed Errors' on the NIC/vmnic, this can be seen after upgrading ESXi hosts from ESXi 7.0U3 to ESXi 8.0U3.
Environment
- VMware vSphere ESXi 8.x, 9.x
- Mellanox pNICs
Cause
- On Mellanox driver 4.22.73.1004 with firmware 14.32.1010, there is a condition, where 'Receive Missed Errors' were not updating and reflecting, specifically packets that were reported as 'out of buffer', on the NIC statistics of the card.
Example is seen below:
NIC statistics for vmnic#: Receive packets dropped: 0 Transmit packets dropped: 0 Total receive errors: 0 Receive length errors: 0 Receive over errors: 0 Receive CRC errors: 0 Receive frame errors: 0 Receive FIFO errors: 0 Receive missed errors: 0
outOfBuffer: 49301854 <<<<
This information is part of the NIC private statistics, which are not displayed in the standard esxcli network nic stats get output.
To view these detailed statistics, you can run the following command on the ESXi host: #/usr/lib/vmware/vm-support/bin/nicinfo.sh | less
- At this time, you may not be receiving any "High pNic error rate detected" notification from vCenter.
- After driver/ firmware upgrade, which can come from ESXi host upgrade, 'out of buffers' are correctly reflected against 'Receive Missed Errors'. This may result in the notification "High pNic error rate detected" being seen on vCenter.
Resolution
To resolve this issue, apply both the recommended driver and firmware updates:
Update ESXi Driver: Install driver 4.23.7.2-1vmw or later (available starting with ESXi 8.0P08). This corrects an issue where "Receive Missed Errors" were misreported as "out of buffers" on the NIC, generating "High pNic error rate detected" warnings in vCenter.
Update Mellanox Firmware: Upgrade the Mellanox adapter firmware to version 26.44.1036 or later to resolve the buffer exhaustion issue.
If the buffer errors continue to increment after applying both updates, contact the hardware OEM/vendor for advanced diagnostic triage.
Additional Information
- For more details on the alarm and its thresholds refer to the following KB:
Alarm about high pNIC error rate being detected - ESXi もしくはドライバをアップグレード後に Mellanox 物理 NIC で "Receive Missed Errors" が検出される
Feedback
