ESXi Host Loses Network Connectivity Due To Undetected Physical Link Failure

Products

• VMware vSphere ESXi

Issue/Introduction

VMware ESXi host(s) experience a loss of network connectivity following a reboot, a failover event or a lack of expected failover event. This issue can potentially affect all hosts in a cluster, though it may currently appear on a subset of hosts.  This also could have been a transient issue, but connectivity issues were seen within vSphere and perhaps logged in vCenter.

Symptoms and environmental keywords include:

• ESXi network utilizes vmnicX and vmnicY which may or may not be VLAN tagging.
• If management is affected, the ESXi host drops off the network and shows as "Not Responding" in the vCenter UI.
• Other vmkernels (vSAN, vMotion, etc) may no longer be working as expected.
• The affected host(s) are actively using vmnicX for the specific vmkernel traffic (can be verified via esxtop), and all network traffic fails when the vmkernel is assigned the affected vmnicX.
• The host does not detect a link DOWN state on vmnicX, preventing a standard failover to vmnicY.
• Manually forcing a failover to vmnicY restores network connectivity and reconnects the host to vCenter.

Cause

The physical network port or upstream switch port connected to vmnicX is misconfigured, encountering a hardware fault, or failing to pass traffic without explicitly dropping the link state. Because the ESXi host still detects an active link on vmnicX, it does not automatically initiate a network failover to the working adapter (vmnicY).

Note that if all the affected hosts are having issues using the same vmnic, and the hosts are all Cisco UCS, there is a possibility that there is a configuration issue with the port profile associated with the specific/affected vmnic.

Resolution

If the Host management vmkernel is currently affected, and the host has lost network connectivity, the following steps can be used to resolve:

1. Connect to the affected ESXi host using the Direct Console User Interface (DCUI) and connect to the host console by pressing Alt-F1. Direct Console User Interface (DCUI) access using ALT+F keys in VMware vSphere ESXi
2. Identify the active vmnic being used for the management network. Identify virtual NIC(vmkernel adapter, VM NIC) and vmnic mapping from esxtop
3. Manually Admin down the problematic network adapter (vmnicX) to force a failover to the secondary adapter (vmnicY).  Setting the admin state up or down for a vmnic interface on ESXi
4. Verify that network connectivity is restored via ping and that the host successfully reconnects to the vCenter Server.
5. Engage the network team and, if necessary, the physical switch hardware vendor to investigate the physical hardware and upstream switch configuration for the port connected to vmnicX and beyond.

If one of the other vmkernels is affected, the above steps can be completed using a standard SSH session or DCUI console.

NOTE: Steps to test ping for specific vmkernels here:  Testing VMkernel network connectivity with the vmkping command

Additional Information

Troubleshooting network and TCP/UDP port connectivity issues on Hosts