Error: Network Connectivity Lost alarm on ESXi hosts

Products

• VMware vSphere ESXi

Issue/Introduction

An ESXi host generates repeated "Network Connectivity Lost" alerts in vCenter Server when a physical uplink (vmnic) experiences intermittent link state changes. This behavior often affects the Management Network, leading to packet drops or host unresponsiveness in vCenter.

• "Network redundancy lost" or "Lost uplink redundancy" alarms appearing in vCenter.
  
  
• Transient packet/ping drops on the management network.
  
  
• Physical management console (e.g., HPE iLO, Dell iDRAC) may show the adapter in a "Degraded" or "Unknown" state.
  
  
• The /var/log/vobd.log records frequent link transitions with Failed criteria: 128.
  DD-MMM-YYYY hh:mm:ss [netCorrelator] [vob.net.pg.uplink.transition.down] Uplink: vmnic# is down. Affected portgroup: Management Network. 0 uplinks up. Failed criteria: 128
DD-MMM-YYYY hh:mm:ss [netCorrelator] [esx.problem.net.connectivity.lost] Lost network connectivity on virtual switch "vSwitch0". Physical NIC vmnic# is down.

  From /var/run/log/vmkernel.log   
DD-MMM-YYYY hh:mm:ss bnxtnet: [vmnic5] NIC Link is down
DD-MMM-YYYY hh:mm:ss bnxtnet: [vmnic5] NIC Link is Up, 10000 Mbps full duplex


• These alarms may be triggered during UPS testing or power maintenance events.

Environment

• ESXi 7.x
• ESXi 8.x
• ESX 9.x

Cause

This alarm triggers if the network driver detects a "link state down" (Criteria 128) caused by an external hardware issue, such as a defective SFP, a bad cable, or a problem with the upstream switchport.

Resolution

To resolve the issue: 

1. Verify the physical connections for the affected vmnic:
   
   • Reseat the network cable at both the host and switch ends.
   • Inspect and replace SFP/SFP+ modules if the issue persists.
2. Monitor vmnic link state:
   
   • Run esxcli network nic stats get -n vmnicX to check for CRC or FIFO errors.
   • If host-side counters are zero, the issue likely resides in the physical cabling or switchport.
3. Hardware Vendor Engagement:
   
   • Check for adapter degradation in the hardware management console (iLO/iDRAC).
   • Contact physical switch vendor for port-level diagnostics when Failed criteria: 128 is seen without host-side driver resets.

If the issue persists, refer to the article “Network adapter (vmnic) is down or fails with a Failed Criteria Code” for additional symptoms and guidance related to Failed criteria 128. 

To determine if the issue is ESXi related, Contact Broadcom support