Total network disconnect on ESXi 8.0 hosts during silent upstream ToR switch failure

Products

• VMware vSphere ESXi

Issue/Introduction

Two or more ESXi 8.0 Update 3 hosts experience a total loss of network connectivity following an upstream network change or physical switch hang (e.g., during an iOS upgrade).

• The hosts remain connected to a secondary, fully functional Top of Rack (ToR) switch.
• Physical links on the ESXi host may initially report as 'Up', resulting in a network blackhole where traffic is pinned to a dead path.
• vCenter reports the affected hosts as 'Not Responding'.
• connectivity is restored immediately once power is removed from the hung physical switch.

Review the vobd.log and vmkernel.log for the following signatures around the time of the outage:

Validate if the host detected a physical link state change at the time of the upstream ToR switch failure:

• Search Target: vobd.log
• Signatures: Search for explicit link down triggers such as "vmnicX linkstate down".
• Example Log Messages:
  In(14) vobd[#####]:  [netCorrelator] #####us: [vob.net.vmnic.linkstate.down] vmnic vmnic1 linkstate down
In(14) vobd[#####]:  [netCorrelator] #####us: [esx.problem.net.vmnic.linkstate.down] Physical NIC vmnic1 linkstate is down
...
In(14) vobd[#####]:  [netCorrelator] #####us: [esx.clear.net.vmnic.linkstate.up] Physical NIC vmnic1 linkstate is up
  
  
• Search Target: vmkernel.log
• Signatures: Search for link down notification corresponding to the affected physical NICs
• Example Log Messsages:
  In(182) vmkernel: cpu55:2098132)bnxtnet: bnxtnet_display_link:####: [vmnic6 : 0x########] NIC Link is down
...
In(182) vmkernel: cpu48:2098132)bnxtnet: bnxtnet_display_link:####: [vmnic6 : 0x########] NIC Link is Up, 10000 Mbps (NRZ) full duplex, Flow control: none

Cause

The failure domain resides at the physical network layer. While the ESXi host eventually detects the link down event and broadcasts failover notifications (RARP) over the healthy secondary link, traffic fails to route if the secondary switch drops these updates or the upstream fabric is logically isolated.

Resolution

If you identify these log signatures coinciding with an upstream switch event, engage your physical network team to investigate the secondary (surviving) switch:

1. Verify if the secondary switch received and processed the RARP/GARP broadcasts from the ESXi host to update its MAC address routing tables.
2. Confirm the switch port connected to the surviving ESXi uplink is in a forwarding state and not in a Spanning Tree Protocol (STP) blocking or learning state.
3. Ensure the secondary switch maintained valid Layer 2 connectivity to the default gateway during the primary switch hang.

Additional Information

For guidance on uplink allocation, see KB 420924 How to Allocate Specific Uplinks to Port Groups on a vSphere Distributed Switch.