Troubleshooting NIC Flapping and Suspected Network Uplink Redundancy Failover Failure

Products

• VMware vSphere ESXi

Issue/Introduction

• This article provides steps for identifying and resolving network uplink redundancy loss caused by intermittent physical interface (pNIC) link flapping on one of the host's pNIC's.
• This issue manifests as repeated link-state changes and data plane packet drops.
• Virtual Machines are temporarily network isolated on a host, even though the redundant uplink pNIC never went DOWN or flapped. 
• Due to that the expectation is that the network traffic would failover to the available redundant uplink which is UP, and therefore there should be no packet loss.
• This leads to the conclusion that failover to the redundant vmnic uplink failed. 

Environment

VMware vSphere ESXi

Cause

• Physical Layer (L1) instability—such as a faulty SFP module, damaged network cable, defective switch port, or a failing physical Network Interface Card (pNIC)—triggers repeated flapping link-state changes.
• It is observed that network loss only happens when the faulty uplink comes UP, but not when it is DOWN, which confirms that failover to the redundant vmnic uplink (pNIC) is working as expected.
• The faulty uplink is technically brought UP from an operational state point of view by the driver, but there is an issue either on the pNIC, cable or physical switch port leading to large TX or RX packet drops.
• Please note the pNIC driver is responsible for bringing the uplink into an UP or DOWN state.
• Once the uplink pNIC is brought UP by the pNIC driver, it will be used by the hypervisor as an available path for forwarding traffic.

Resolution

1. Analyze vmkernel.log for recurring NIC link state event patterns like the following:
   • In(182) vmkernel: cpu20:2098114)netschedHClk: NetSchedHClkNotify:4749: vmnicX: link down notification
   • In(182) vmkernel: cpu20:2098114)netschedHClk: NetSchedHClkNotify:4741: vmnicX: link up notification
2. Review hostd logs for error statistics on the faulty flapping vmnic uplink:
   • Search for droppedRx, droppedTx, and errorsRx increments like the example below. High and increasing error counts confirm physical path degradation over the flapping uplink:
     Wa(164) Hostd[2099202]: [Originator@6876 sub=Statssvc.StatsCollector] Error stats for pnic: vmnic0
Wa(164) Hostd[2099183]: --> droppedRx: 97045
Wa(164) Hostd[2099183]: --> droppedTx: 10214299
Wa(164) Hostd[2099183]: --> errorsRx: 14
      
3. Replace the physical cable and SFP module associated with the affected vmnic.
4. Move the connection to a different physical switch port.
5. If link instability and hardware errors persist, initiate a replacement of the physical Network Interface Card (pNIC).
6. Engage with your hardware vendor if the above steps do not resolve the issue.

Additional Information

• Contact Broadcom Support
• Creating and managing Broadcom support cases