Receive Missed Errors detected on Mellanox pNICs after updating the driver or ESXi
Article ID: 393164
Updated On:
Products
Issue/Introduction
- After upgrading Mellanox driver NICs, you may see "High pNIC Error rate detected" within vCenter shortly after.
- Inspecting further from esxcli console, inputting below command:
#esxcli network nic stats get -n vmnic# - You may see numerous 'Receive Missed Errors' on the NIC/vmnic, this can be seen after upgrading ESXi hosts from ESXi 7.0U3 to ESXi 8.0U3.
Environment
- VMware vSphere ESXi 8.x, 9.x
- Mellanox pNICs
Cause
- On Mellanox driver 4.22.73.1004 with firmware 14.32.1010, there is a condition, where 'Receive Missed Errors' were not updating and reflecting, specifically packets that were reported as 'out of buffer', on the NIC statistics of the card.
Example is seen below:
NIC statistics for vmnic#:
Receive packets dropped: 0
Transmit packets dropped: 0
Total receive errors: 0
Receive length errors: 0
Receive over errors: 0
Receive CRC errors: 0
Receive frame errors: 0
Receive FIFO errors: 0
Receive missed errors: 0
outOfBuffer: 49301854 <<<<
This information is part of the NIC private statistics, which are not displayed in the standard esxcli network nic stats get output.
To view these detailed statistics, you can run the following command on the ESXi host: #/usr/lib/vmware/vm-support/bin/nicinfo.sh | less
- At this time, you may not be receiving any "High pNic error rate detected" notification from vCenter.
- After driver/ firmware upgrade, which can come from ESXi host upgrade, 'out of buffers' are correctly reflected against 'Receive Missed Errors'. This may result in the notification "High pNic error rate detected" being seen on vCenter.
Resolution
This has since been addressed and corrected, with driver 4.23.6.2 and firmware 14.32.1900, where the 'Receive Missed Errors' are correctly reflecting the 'out of buffers' being reported on the NIC.Which is leading to "High pNic error rate detected" being seen on vCenter.
In order to rectify the 'out of buffers' or 'Receive Missed Errors', this will need to be triaged with the vendor of the card.
Additional Information
- For more details on the alarm and its thresholds refer to the following KB:
Alarm about high pNIC error rate being detected - ESXi もしくはドライバをアップグレード後に Mellanox 物理 NIC で "Receive Missed Errors" が検出される
Feedback
