ESXi host displays "Uplink redundancy lost" alert in vCenter due to CRC errors on the NIC

Products

• VMware vSphere ESXi

Issue/Introduction

• Receiving the following alerts in vCenter on the ESXi host:  "Lost uplink redundancy" and/or "Network uplink redundancy lost"
• Confirmed that NIC driver/firmware is compatible as per VMware HCL: Determining Network/Storage firmware and driver version in ESXi
• Checking NIC stats, it shows that there are CRC error increasing rapidly:
  • Login to ESXi host over SSH session and run the following command:  esxcli network nic stats get -n vmnic<X> 
    Example output:
    

    [root@ESXi:~] esxcli network nic stats get -n vmnic0
    NIC statistics for vmnic0
       Packets received: 599736627
       Packets sent: 489239060
       Bytes received: 189333612000
       Bytes sent: 1264126208861
       Receive packets dropped: 0
       Transmit packets dropped: 0
       Multicast packets received: 95624657
       Broadcast packets received: 13147021
       Multicast packets sent: 760144
       Broadcast packets sent: 4614094
       Total receive errors:2349184
       Receive length errors: 0
       Receive over errors: 0
       Receive CRC errors: 2349184 <<<<<<<-------- CRC errors increasing rapidly
       Receive frame errors: 0
       Receive FIFO errors: 0
       Receive missed errors: 0
       Total transmit errors: 0
       Transmit aborted errors: 0
       Transmit carrier errors: 0
       Transmit FIFO errors: 0
       Transmit heartbeat errors: 0
       Transmit window errors: 0

Cause

• A Cyclic Redundancy Check (CRC) is an error-detecting code commonly used in digital networks and storage devices to detect accidental changes to digital data.
• CRC error: FCS (Frame Check Sequence) field contains a 4-byte CRC value used for error checking.
• When a source host assembles a packet, it performs a CRC calculation on all fields in the packet except the Preamble, SFD (Start Frame Delimiter), and FCS using a predetermined algorithm.
  The source host stores the value in the FCS field and transmits it as part of the packet.
  When the packet is received by the destination host, it performs a CRC test again by using the same algorithm.
  If the CRC value calculated at the destination host does not match the value in the FCS field, the destination host discards the packet, considering this as a CRC Error.

Resolution

• Please reach to respective hardware vendors(Physical Switch/SFT/Cable/Network Adapter) to troubleshoot further.
• Common troubleshooting checks that can be done:
  • Check if SFP is compatible with card and similar to the other vmnic's
  • Check and clean transceiver optic part
  • Check if SFP is correctly seated on the card and physical switch switchport
  • Verify cable path for damage or with tight bends/short radius that may increase cable attenuation for a fiber type.
  • Verify physical switchport stats for any transmit errors

Additional Information

Virtual Machines network interruption with “Receive CRC errors” statistics increased rapidly
Network adapter (vmnic) is down or fails with a Failed Criteria Code
Physical Link Down with Redundancy Degraded
Troubleshooting NIC errors and other network traffic faults in ESXi