ESXi hosts are not responding and appear grayed out in vCenter

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

This article provides troubleshooting steps to determine why a host is inaccessible from the vSphere Client.

Host enters a "Not Responding" state in vCenter Server
No Purple Screen of Death (PSOD) or core dump is generated.
Unable to connect to the ESXi host directly using vSphere Client.
All the virtual machines registered to the ESXi host are grayed out.
All hypervisor logging (vmkernel.log and hostd.log in ESXi - /var/log) abruptly stops with no preceding errors or warnings.
iLO/AHS logs show System Management Interrupt (SMI) spikes or no hardware alerts despite the freeze.
Total loss of management connectivity (Ping, SSH, and DCUI) requiring a physical power cycle to recover.
The vpxd.log files residing in the vCenter Server (/var/log/vmware/vpxd/) contain events indicating an error when attempting to communicate with the host. The events always contain the words vmodl.fault.HostCommunication and appear similar to the following:
[VpxLRO] -- ERROR task-internal-6433833 -- host-##### -- vim.host.NetworkSystem.queryNetworkHint: vmodl.fault.HostCommunication:
(vmodl.fault.HostCommunication) {
dynamicType = <unset>,
faultCause = (vmodl.MethodFault) null,
msg = "",
}
[VpxdMoHost::CollectRemote] Stats collection cannot proceed because host may no longer be available or reachable: vmodl.fault.HostCommunication.

The issue appears on multiple hosts. Note the opID that identifies the offending host:
[YYYY-MM-DDTHH:MM:SS][29348 verbose 'Default' opID=f6a80d55] [ServerAccess] Attempting to connect to service at vc1.####:10443
If this issue occurs due to a communication error between the host and the vCenter Server, but the host is still responsive to user interaction, events similar to these appear on ESXi - /var/log/vmware/vpxa.log files:
Failed to bind heartbeat socket (-1). Using any IP.
Agent can't send heartbeats.msg size: 66, sendto() returned: Network is unreachable.

For instructions on retrieving logs and log file locations, see Location of vCenter Server log files.

Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0
VMware vCenter Server 7.0
VMware vCenter Server 8.0

Cause

The following are possible causes, but are not limited to:

External networking changes or network connectivity issues.
Host hardware-related issues, including underlying physical hardware or firmware faults causing a chassis-level lockup.
On HPE Gen11 Intel-based systems, this frequently correlates with high System Management Interrupt (SMI) rates or System ROM versions prior to 2.80.

Resolution

To determine why a host is inaccessible and to restore service, follow these steps:

Verify the current state of the host hardware and power.
1. Physically inspect the hardware and note any lights on the server face indicating power or hardware status.
2. If the hardware lights indicate an issue, consult the hardware vendor documentation or support.
3. If the hardware is currently turned off, turn it on and refer to Unexpected ESXi Reboot or Shutdown.
  
  Note: Depending on the physical environment configuration, consider connecting to the physical host using a remote hardware interface provided by the hardware vendor. Consult the hardware vendor documentation on how this interface interprets hardware conditions.
Determine the state of the user interface at the physical console.
1. If the user interface does not respond to user interaction, refer to Determining why an ESX/ESXi host does not respond to user interaction at the console.
2. If the user interface displays a purple diagnostic screen, refer to Interpreting a host purple diagnostic screen.
  
  Note: Remote applications (such as a keyboard/video/mouse switch or a remote hardware interface) are known to interfere with keyboard and mouse functionality. Verify responsiveness at the local physical console before taking action.
Test management network connectivity.
1. Determine if the host responds to ping. Menu options are provided in the direct console to test the management network.
2. If the host responds to user interaction but does not respond to pings, a networking issue might exist. Refer to Intermittent network connectivity on ESXi or host disconnect issues.
Verify ESXi host vSphere client connectivity.
1. Open the ESXi host vSphere client.
2. Specify the hostname or IP address of the host, along with the appropriate credentials for the root user, and click login.
3. If a connection failure error occurs, the agents responsible for facilitating the vSphere API might not be functioning. Refer to Diagnosing the vSphere Client when it fails to connect to an ESXi host or vCenter Server or review logs for the error "Agent can't send heartbeats because socket address structure initialization is failing".
4. If the direct connection succeeds, but it continues to show as unresponsive from vCenter Server, verify if the correct Managed IP Address is set in vCenter Server. Refer to Troubleshooting an ESXi host in a "not responding"/"disconnected" state.
Determine if the host rebooted.
1. Physically log into the console of the host.
2. Type the command uptime to view the uptime of the host. If the host recently rebooted, refer to Unexpected ESXi Reboot or Shutdown.
Isolate the failure domain and remediate hardware lockups.
1. Isolate the failure domain. If the steps above confirm the absence of hypervisor-level triggers (e.g., storage APD/PDL or resource exhaustion), an underlying hardware lockup may be occurring.
2. Vendor Analysis: export and provide Active Health System (AHS) and iLO logs covering the incident window to the hardware vendor for root cause analysis.

Additional Information

High Availability Triggers

The High Availability (HA) feature uses different triggers than vCenter Server to ensure a host is operational.

vCenter Server Alarm: The Host connection and power state alarm triggers due to a vmodl.fault.HostCommunication fault. This fault occurs if vCenter Server cannot communicate with a host using the vSphere API.
HA Isolation Response: This response triggers when a host agent cannot communicate with agents on other hosts and fails to reach the designated isolation address (the default gateway, by default). When both conditions are met, the host executes the configured HA isolation response.

Different agents manage both systems and communicate with different network hosts. Consequently, the relationship between these states is as follows:

A host showing as Not Responding in vCenter Server does not always trigger an HA isolation response. It might maintain a network connection with other hosts or its isolation address, remaining non-isolated.
A host experiencing an HA isolation response typically appears as Not Responding in vCenter Server.

References: