ESXi hosts are not responding and appear grayed out in vCenter

VMware vCenter Server VMware vSphere ESXi

This article provides troubleshooting steps to determine why a host is inaccessible from the vSphere Client.

Symptoms:

Host enters a "Not Responding" state in vCenter Server.
No Purple Screen of Death (PSOD) or core dump is generated.
Unable to connect to the ESXi host directly using vSphere Client.
All the virtual machines registered to the ESXi host are grayed out.
All hypervisor logging (vmkernel.log, hostd.log) abruptly stops with no preceding errors or warnings.
iLO/AHS logs show System Management Interrupt (SMI) spikes or no hardware alerts despite the freeze.
Total loss of management connectivity (Ping, SSH, and DCUI) requiring a physical power cycle to recover.
The vpxd.log files residing in the vCenter Server contain events indicating an error when attempting to communicate with the host. The events always contain the words vmodl.fault.HostCommunication and appear similar to the following:
[VpxLRO] -- ERROR task-internal-6433833 -- host-##### -- vim.host.NetworkSystem.queryNetworkHint: vmodl.fault.HostCommunication:
(vmodl.fault.HostCommunication) {
dynamicType = <unset>,
faultCause = (vmodl.MethodFault) null,
msg = "",
}
[VpxdMoHost::CollectRemote] Stats collection cannot proceed because host may no longer be available or reachable: vmodl.fault.HostCommunication.

The issue appears on multiple hosts. Note the opID that identifies the offending host:
[YYYY-MM-DDTHH:MM:SS][29348 verbose 'Default' opID=f6a80d55] [ServerAccess] Attempting to connect to service at vc1.####:10443
If this issue occurs due to a communication error between the host and the vCenter Server, but the host is still responsive to user interaction, events similar to these appear in the /var/log/vmware/vpxa.log files:
Failed to bind heartbeat socket (-1). Using any IP.
Agent can't send heartbeats.msg size: 66, sendto() returned: Network is unreachable.

For instructions on retrieving logs and log file locations, see Location of vCenter Server log files.

External networking changes or network connectivity issues.
Host hardware-related issues, including underlying physical hardware or firmware faults causing a chassis-level lockup.
On HPE Gen11 Intel-based systems, this frequently correlates with high System Management Interrupt (SMI) rates or System ROM versions prior to 2.80.

To determine why a host is inaccessible and restore service, follow these steps:

1. Verify the current state of the host hardware and power.

Physically inspect the hardware and note any lights on the server face indicating power or hardware status.
If the hardware lights indicate an issue, consult the hardware vendor documentation or support.
If the hardware is currently turned off, turn it on and see Unexpected ESXi Reboot or Shutdown.

Note: Depending on the physical environment configuration, consider connecting to the physical host using a remote hardware interface provided by the hardware vendor. Consult the hardware vendor documentation on how this interface interprets hardware conditions.

2. Determine the state of the user interface at the physical console.

If the user interface does not respond to user interaction, see Determining why an ESX/ESXi host does not respond to user interaction at the console (341047).
If the user interface displays a purple diagnostic screen, see Interpreting a host purple diagnostic screen.

Note: Remote applications (such as a Keyboard/Video/Mouse switch or a remote hardware interface) are known to interfere with keyboard and mouse functionality. Verify responsiveness at the local physical console before taking action.

3. Test management network connectivity.

Determine if the host responds to ping. Menu options are provided in the direct console to test the management network.
If the host responds to user interaction but does not respond to pings, a networking issue might exist. Review the following resources: Intermittent network connectivity on ESXi or host disconnect issues

4. Verify vSphere Client connectivity.

Open the vSphere Client.
Specify the hostname or IP address of the host, along with the appropriate credentials for the root user, and click Login.
If a connection failure error occurs, the agents responsible for facilitating the vSphere API might not be functioning. See Diagnosing the vSphere Client when it fails to connect to an ESXi host or vCenter Server or review logs for the error "Agent can't send heartbeats because socket address structure initialization is failing".
If the direct connection succeeds, but it continues to show as unresponsive from vCenter Server, verify if the correct Managed IP Address is set in vCenter Server. See Troubleshooting an ESXi host in a "not responding"/"disconnected" state.

5. Determine if the host rebooted.

Physically log in to the console of the host.
Type the command uptime to view the uptime of the host. If the host recently rebooted, see Unexpected ESXi Reboot or Shutdown.

6. Isolate the failure domain and remediate hardware lockups.

Isolate the Failure Domain: If the steps above confirm the absence of hypervisor-level triggers (e.g., storage APD/PDL or resource exhaustion), an underlying hardware lockup may be occurring.
Vendor Analysis: Export and provide Active Health System (AHS) and iLO logs covering the incident window to the hardware vendor for root cause analysis.

High Availability Triggers

The High Availability (HA) feature uses different triggers than vCenter Server to ensure a host is operational.

vCenter Server Alarm: The Host connection and power state alarm triggers due to a vmodl.fault.HostCommunication fault. This fault occurs if vCenter Server cannot communicate with a host using the vSphere API.
HA Isolation Response: This response triggers when a host agent cannot communicate with agents on other hosts and fails to reach the designated isolation address (the default gateway, by default). When both conditions are met, the host executes the configured HA isolation response.

Different agents manage both systems and communicate with different network hosts. Consequently, the relationship between these states is as follows:

A host showing as Not Responding in vCenter Server does not always trigger an HA isolation response. It might maintain a network connection with other hosts or its isolation address, remaining non-isolated.
A host experiencing an HA isolation response typically appears as Not Responding in vCenter Server.

References:

Impact/Risks: None