Network connectivity loss during physical switch maintenance in VMware vSphere ESXi

Products

• VMware vSphere ESXi

Issue/Introduction

Issue: During physical network maintenance or switch replacement activities, virtual machines lose network connectivity even though redundant physical uplinks are present.

• vCenter reports multiple physical NIC link-state changes.
• Uplink redundancy alarms trigger and clear ('Uplink redundancy restored') within seconds.
• VMs experience a total network outage.
• ESXi vobd.log may not show persistent NIC-down events because the physical link remains active at the electrical level.

From the host's /var/run/log.vmkernel.log, investigate for "NIC Link is down" events. Example:

<timestamp> In(182) vmkernel: cpu31:2098697)bnxtnet: bnxtnet_display_link:2923: [vmnic2 : 0x4523c42e0000] NIC Link is down

Environment

• VMware ESXi 7.x/8.x
• Cisco UCS Manager / Fabric Interconnects

Cause

The issue occurs when a physical switch is removed from management configuration (e.g., UCS Manager) or powered down without first administratively shutting down the ports. The ESXi host continues to see a 'Link Up' status due to residual electrical signaling or keep-alives from the switch hardware, preventing the hypervisor from triggering a failover to the healthy redundant uplink.

 

Resolution

To ensure successful failover during maintenance:

1. Log in to the physical switch management interface or UCS Manager.
2. Administratively shut down (Admin Down) all physical ports connected to the ESXi host uplinks that will be undergoing maintenance.
3. Verify the failover in ESXi using one of the following methods:
   • Run esxtop, press n, and ensure the VMs have migrated to the surviving vmnic.
   • Perform a packet capture on the active uplink: pktcap-uw --uplink vmnicX --dir 2 -o -| tcpdump-uw -enr -
4. Once maintenance is complete, re-enable the ports and verify the "Redundancy Restored" alarm in vCenter.

Note:You can utilize LACP or Beacon Probing to detect "grey failures" where a link is electrically up but logically dead.

For instructions on collecting logs, see Collecting diagnostic information for VMware ESXi. To speak with a customer representative, see Contact Support.

 

Additional Information

• Understanding IP Hash load balancing
• While using LACP, and in environments configured with Slow LACP timers (30-second transmission intervals, 90-second timeout), the upstream physical switch may continue to route return traffic to the failed link for up to 90 seconds. This can be prevented by following the above resolution to minimize the packet drop. Reference: vSAN VMs Restart Unexpectedly During Single NIC Failure in LACP LAG Due to Timer Mismatch
• For more information on LACP timers, refer: LACP Timeout mode parameter mismatch between the ESXi host and physical switches