Mellanox nmlx5_core NICs Experience High Physical CRC Errors From Degraded Switch Ports

Products

• VMware vSAN
• VMware vSphere ESXi

Issue/Introduction

• ESXi hosts equipped with Mellanox Technologies ConnectX-6 Lx network adapters utilizing the nmlx5_core driver, experience packet drops, throughput degradation.
• The observed packet drops can also lead to the vSAN health alarm, "High pNic error rate detected".
• Validating the private driver counters for the concerned vmnic, reveals an active, rapid accumulation of physical layer frame check sequence errors and lane symbol errors.
• Specifically counters: rxCrcErrorsPhy and rxSymbolErrorsPhy are observed to be increasing in a given time frame.
• Command outputs reports the following health state:

usr/lib/vmware/vm-support/bin/nicinfo.sh: 

NIC: vmnic0
   rxCrcErrorsPhy: 290127
   rxSymbolErrorsPhy: 290108
   rxErrLane_0_Phy: 72531
   MAC Address: <REDACTED_MAC_ADDRESSES>
   PCI Device: <REDACTED_SECRETS>

Environment

VMware vSphere ESXi

VMware vSAN

Mellanox Technologies ConnectX-6 Lx / Dx Adapters

Cause

Defective or degraded upstream physical network switch ports causing physical layer (Layer 1) signal attenuation and data corruption prior to host ingestion.

Resolution

1. Verify the current driver and firmware versions against the Broadcom's Hardware Compatibility guide: HCL
   • For more information on how to validate driver/firmware version for adapters, please refer article: Determining Network/Storage firmware and driver version in ESXi
2. To validate the error growth rate by running on the ESXi host within a time frame, please run the below watch command and observe the increase in the adapter's error stats: 
   • watch -d esxcli network nic stats get -n vmnicX
3. Schedule a maintenance window and place the affected ESXi host into Maintenance Mode via vSphere Client or SDDC Manager.
   • Physically migrate the fiber optic patch cabling and transceivers from the degraded switch ports to known functional alternative ports on the upstream network switch.
   • Coordinate with the network engineering team to administratively disable (shutdown) and flag the defective switch ports to prevent future workload allocations.
   • Clear the host hardware interface statistics.
4. Reboot the ESXi host to clear the physical hardware counters, as these statistics cannot be reset live.
5. Monitor the adapter statistics for 24 hours to ensure "rxCrcErrorsPhy" remains at zero.

 

Additional Information

For more information please refer the below articles: 

• Receive length errors detected on Mellanox nmlx5_core NICs.
• High pNic error rate detected on Mellanox network adapter.
• Seeing "High pNIC error rate" vSAN alarm on vSAN Hosts with Mellanox NIC Cards