ESXi hosts show as "Not Responding" in vCenter and management access via SSH fails

Products

• VMware vSphere ESXi

Issue/Introduction

•  Multiple ESXi hosts show as "Not Responding" or "Disconnected" in vCenter Server.
•  Virtual Machines (VMs) on affected hosts may appear as "Inaccessible" or "Invalid" after a host reboot.
•  The ESXi host is reachable via out-of-band management (IPMI/iDRAC) but unreachable via the management network.
•  Pinging the default gateway from the ESXi DCUI fails.
•  Physical NICs (vmnics) show a status of "Link Up", even though network traffic is not passing to the core.

Environment

• VMware vSphere ESXi 7.x
• VMware vSphere ESXi 8.x
• VCF (VMware Cloud Foundation)

Cause

• This issue occurs when an upstream physical switch loses its connection to the spine or core network, but the physical link between the ESXi host and the access switch remains active.
• In a default configuration using “Link Status only” for failover detection, the ESXi host only monitors the physical state of its own NIC and the immediate switch port.
• If the link remains "Up," the host continues to hash traffic to that uplink based on the load-balancing algorithm.
• If the upstream network is isolated, packets are dropped at the access switch, this can lead to a loss of management heartbeats and vCenter disconnection.

 

• If the management traffic is routed to the link attached to SW 2 the host will become disconnected as it has no path to the Core switch.
• The uplink choice is a matter of the hashing algorithm.

Resolution

Identifying the Fault

1.  Identify the affected uplink by performing a `vmkping` to the gateway while administratively shutting down one vmnic at a time:
   esxcli network nic down -n vmnicX
2. Once the faulty path is identified, keep the problematic uplink down to force traffic through the functional path.
    
3. Engage your physical network engineering team to inspect upstream switch ports, spine-leaf connectivity, and VLAN tagging on the affected switch.

Confirming the Fault and where the ICMP Reply Stops.

• To confirm the issue as outside of the host or hosts, a packet captures of ICMP pings can be done.
• The captures will follow the packets from the source to the destination and back.
• This will require the network administrator to perform captures on the physical network data path.
• This KB is the reference for use of the ESX host's native packet capture function. Using the pktcap-uw tool in ESXi
  •  Note the ICMP request leaves the host’s uplink.
  •  Note if the ICMP reply is received or not.
  •  The ICMP reply is not seen in this case. The involvement of the network administrator will be necessary to chase where the ICMP request gets to and stops. 
  •  Then follow the ICMP reply, if generated, back to where it stops on the way back from the destination(vCenter). This is where the issue lies.