ESXi hosts do not respond and is grayed out in vCenter
search cancel

ESXi hosts do not respond and is grayed out in vCenter

book

Article ID: 345273

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

This article provides troubleshooting steps to determine why an ESXi host is inaccessible from the vSphere Client.

Symptoms:The 
  • ESXi host is not responding to the vCenter Server
  • All virtual machines that are registered to the  ESXi host are grayed out.
  • You are unable to connect to the ESXi host directly using vSphere Client.
  • The vpxd.log files residing in the vCenter Server may contain events indicating an error when attempting to communicate with the ESXi host. The events always contain the words vmodl.fault.HostCommunication may appear similar to the following examples:

    [VpxLRO] -- ERROR task-internal-6433833 -- host-24499 -- vim.host.NetworkSystem.queryNetworkHint: vmodl.fault.HostCommunication:
    (vmodl.fault.HostCommunication) {
    dynamicType = <unset>,
    faultCause = (vmodl.MethodFault) null,
    msg = "",
    }

    [VpxdMoHost::CollectRemote] Stats collection cannot proceed because host may no longer be available or reachable: vmodl.fault.HostCommunication.

    For more information on the location of the vpxd.log file, see.

Location of vCenter Server log files


The issue may appear on multiple hosts, keep note on the opID that identifies the offending ESXi host:

[YYYY-MM-DDTHH:MM:SS][29348 verbose 'Default' opID=f6a80d55] [ServerAccess] Attempting to connect to service at vc1.hostname.vmware.net:10443

 
  • If this issue occurs due to a communication issue between the ESXi host and the vCenter Server, but the host is still responsive to user interaction, you may see events similar to these in the /var/log/vmware/vpxa.log files:

    Failed to bind heartbeat socket (-1). Using any IP.

    Agent can't send heartbeats.msg size: 66, sendto() returned: Network is unreachable.


Environment

VMware vSphere ESXi 7.0
VMware vSphere ESXi 8.0
VMware vCenter Server 7.0
VMware vCenter Server 8.0 

Cause

  • External networking changes.
  • Networking issue
  • ESXi host hardware related
  • Storage, hostd or other services related issues. 

Resolution

To determine why an ESXi host is inaccessible:

  1. Verify the current state of the ESXi host hardware and power. Physically go to the ESXi host hardware, and make note of any lights on the face of the server hardware that may indicate the power or hardware status. For more information regarding the hardware lights, consult the hardware vendor documentation or support.

    Note: Depending on the configuration of your physical environment, you may consider connecting to the physical host by using a remote hardware interface provided by your hardware vendor. For more information about how this interface interprets the condition of the hardware, consult the hardware vendor documentation or support.
     
    • If the hardware lights indicate that there is a hardware issue, consult the hardware vendor documentation or support to identify any existing hardware issues.
    • If the hardware is currently turned off, turn it on and see Unexpected ESXi Reboot or Shutdown.
  2. Determine the state of the user interface of the ESXi host in the physical console.

    Note: Depending on the configuration of your physical environment, you may consider connecting to the physical host by using a remote application such as a Keyboard/Video/Mouse switch or a remote hardware interface provided by your hardware vendor. These interfaces are known to interfere with keyboard and mouse functionality. VMware recommends verifying the responsiveness at the local physical console before taking any action.

     
  4. Determine if the ESXi host responds to ping responses. Several menu options are provided to test the management network. If the ESXi host responds to user interaction but does not respond to pings, you may have a networking issue. For more information, see Intermittent network connectivity on ESXi or host disconnect issues.
     
  5. Verify that you can connect to the VMware ESXi host using vSphere Client:
     
    1. Open the vSphere Client.
    2. Specify the hostname or IP address of the VMware ESXi host, along with the appropriate credentials for the root user.
    3. Click Login.
       
  6. Determine if the ESXi host is rebooted.
     
    1. Physically log in to the console of the ESXi host.
    2. Type the command uptime to view the uptime of the ESXi host. If the ESXi host is recently rebooted, see Unexpected ESXi Reboot or Shutdown.



Additional Information

 

High Availability
 
High Availability (HA) feature uses a different trigger than vCenter Server when ensuring that an  ESXi is operational. The following is a brief explanation of each criteria:
 
  • The Host connection and power state alarm is triggered as a result of a HostCommunication fault. A HostCommunication fault occurs if vCenter Server is unable to communicate to an  ESXi host using the vSphere API.
  • The HA isolation response is triggered as a result of an agent on the ESXi host that is unable to communicate with agents on other ESXi hosts (not the vCenter server). It must also fail to communicate with a designated isolation address (by default, it is the default gateway). If both of these conditions are met, the host performs the designated HA isolation response.

Both systems are managed by different agents and may communicate with different hosts on the network. Therefore, with respect to the relationship:

  • A host that is not responding within vCenter Server does not always trigger a high availability isolation response. It may still be maintaining a network connection with other hosts or its isolation address and thus is not isolated .
  • A host experiencing an HA isolation response is likely to appear as Not Responding within vCenter Server.

Troubleshooting an ESXi host in a "not responding"/"disconnected" state
Testing network connectivity with the ping command
Identifying issues with and setting up DNS name resolution on ESXi Server
Diagnosing the vSphere Client when it fails to connect to an ESXi host or vCenter Server
Intermittent network connectivity on ESXi or host disconnect issues
Interpreting a host purple diagnostic screen
Configuring VMware ESXi Management Network from the direct console
Determining why an ESXi host does not respond to user interaction at the console
Unexpected ESXi Reboot or Shutdown
Location of log files for Broadcom products
Reconnecting or adding an ESXi host to vCenter fails with the error: "Agent can't send heartbeats because socket address structure initialization is failing"

Impact/Risks:
None

Powered by Wolken Software