High pNic error rate detected due to high "Receive length errors" on host uplinks due to driver issue

Products

• VMware vSphere ESXi

Issue/Introduction

ESX host(s) experience network performance degradation characterized by high counts of "Receive length errors" and "Total receive errors" on physical interfaces (vmnics).

• "High pNic error rate detected. Check the host's vSAN performance view for details" displays in vCenter for one or more ESXi hosts.
  
  
• May observe network packet drops at the ESXi host level.
  
  
• NIC statistics show increasing Receive length errors counter values.
  
  
• May impact multiple hosts within the environment simultaneously.
  
  
• The vmnic(s) driver is either not present in the VMware by Broadcom Compatibility Guide or is not at the latest bug-fix release listed.

Environment

VMware vSphere ESXi

Cause

The ESXi host(s)s are running an outdated and/or uncertified driver and firmware combination for the ESXi version and build.

This specific versioning is not present on the VMware by Broadcom Compatibility Guide for the ESXi version and build.

Running uncertified or outdated driver/firmware stacks can lead to hardware-to-software communication faults, specifically it may cause the driver to misinterpret packet lengths or fail to process receive buffers correctly.

Resolution

1. Identify the current NIC driver and firmware in use using the ESXi CLI:
   
   esxcli network nic get -n vmnicN | grep -C1 Firmware
   
   where N is the vmnic number
   
   Example output for a device that uses the qedentv device driver:
   
   Driver: qedentv          
Firmware Version: mfw 8.65.4.0 storm 8.72.1.0          
Version: 3.71.80.0

2. Identify the search parameters for lookup in the VMware by Broadcom Compatibility Guide:
   
   vmkchdev -1 | grep vmnic
   
   Example output:
   
   0000:10:00.0 1077:8070 1077:0068
   
   

   • VID = Vendor ID (e.g 1077)

      

   • DID = Device ID (e.g. 8070)

      

   • SVID = Sub-Vendor ID (1077)

   • Max SSID = Sub-Vendor Device ID (0068)
     
     

3. In a browser, open VMware by Broadcom Compatibility Guide and select I/O Devices under Platform & Compute
   
   
4. Scroll down in the filters on the left-hand side until you reach the VID filter.
   
   
   
5. Enter the VID, DID, SVID, Max SSID identified above and then scroll back up and click View Results
   
   This will normally produce one or at most a few matches. Right-click on the model number link to open it in a new window.
   
   
   
6. Under ESXi versions, select the appropriate release that you have from the drop-down.
   
   This will produce a list of all the versions of the driver that the vendor has informed VMware that it is certified.
   
   Note: the firmware version listed is the firmware that was present at the time of the certification test, not necessarily the latest bug-fix release or the recommended one.
   
   
   
7. If your current driver is not the most recent driver on the list or is not on the list at all, you should contact the vendor to obtain the latest bug-fix release for the driver and firmware.
   
   Note: In the KB 318542 FAQ: Recommendation for drivers/firmware for ESXi hosts we define who we mean by vendor.
   
   
   
8. Follow the instructions in KB 343324 Download and install async drivers in VMware ESXi to update on your affected host(s)