Troubleshooting an ESXi host in a "not responding"/"disconnected" state
search cancel

Troubleshooting an ESXi host in a "not responding"/"disconnected" state

book

Article ID: 344682

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi VMware vCenter Server 6.0 VMware vCenter Server 7.0 VMware vCenter Server 8.0 VMware vSphere ESXi 6.0 VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0

Issue/Introduction

This article provides steps to troubleshoot when an ESXi is in disconnected or not responding state in vCenter Server. In addition, it provides steps to help eliminate common causes by verifying the network configuration and management server agents are correct and confirm the availability of resources on each ESXi host.

  • An ESXi host shows as "not responding" in vCenter Server.
    • On the vCenter UI --> ESXI host --> Monitor --> Event, the message: "Cannot synchronize host <fqdn>".
  • An ESXi host shows as disconnected in vCenter Server.
  • Cannot connect the ESXi host to vCenter Server.
  • Virtual machines on an ESXi host show as greyed out in vCenter Server.
  • While accessing Direct Console (DCUI) and if we toggle between DCUI console and banner screen(For more information please follow Accessing DCUI/Console of ESXi using ALT+F Keys)  we get Realtime errors of hostd agent as below. "hostd detected to be non-responsive"

  • When attempting to add an ESXi to vCenter Server, the following errors may be visible:

    Unable to access the specified host, either it doesn't exist, the server software is not responding, or there is a network problem
     
  • In the vpxd.log file:

<YYYY-MM-DD>T<time>.696-05:00 [08128 info 'vpxdvpxdMoHost' opID=########-########-##] [HostMo] host connection state changed to [DISCONNECTED] for host-ID
<YYYY-MM-DD>T<time>.508-04:00 [04944 error 'vpxdvpxdInvtHostCnx' opID=HB-host-ID@####-########] [VpxdInvtHostSyncHostLRO] FixNotRespondingHost failed for host host-ID, marking host as notResponding
<YYYY-MM-DD>T<time>.633-04:00 [00812 error 'vpxdvpxdInvtHostCnx' opID=HB-host-ID@####-########] [VpxdInvtHostSyncHostLRO] FixNotRespondingHost failed for host host-ID, marking host as notResponding


For more information, see Location of vCenter Server log files

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on environment.

  • Unbale to ping the default gateway of the ESXi host from DCUI when the ESXi host is in disconnected state as shown below:

 

Environment

VMware vCenter Server 7.x

VMware vCenter Server 8.x

Cause

There can be multiple causes of ESXI host being in Not-Responding/Disconnected state.

For Example:

  • If hostd agent itself is unresponsive.(Hostd agent can be unresponsive due to multiple underlying factors that can be further investigated through ESXI logs)
  • Network issues.(Ping loss, gateway reachability issue, LACP(lag) configuration issue, management network connectivity issue)
  • Storage issues.

 

Resolution

Important: Before proceeding review KB Article Understanding the difference between "Not Responding" and "Disconnected" ESXi hosts in VMware vCenter Server (2121031).

As there are a number of reasons why the ESXi host reaches a “Not Responding” state, VMware strongly recommends to:

  • Validate each troubleshooting step below.
  • Each step provides instructions or a link to another Knowledge Base article to eliminate possible causes and corrective action to resolve issue.  

Note: The steps are ordered in the most appropriate sequence to isolate the issue and identify the proper resolution. After each step is completed, try to reconnect the ESXi host back to vCenter Server. Please do not skip a step.

VMware ESXi host that is in a "Not Responding" state

ESXi

  1. Verify that the ESXi host is accessible from vCenter server or vSphere Client.For more information, see ESX/ESXi hosts do not respond and is grayed out 
  2. Verify that the ESXi host can be reconnected, or if reconnecting the ESXi host resolves the issue. For more information, see Changing an ESXi or ESX host's connection status in vCenter Server.
  3. Verify that the ESXi host is able to respond back to vCenter Server at the correct IP address. If vCenter Server does not receive heartbeats from the ESXi host, it goes into a not responding state. To verify if the correct managed IP Address is set, see Verifying the vCenter Server Managed IP Address and  See also, ESXi host disconnects from vCenter Server after adding or connecting it to the inventory and ESXi host keeps disconnecting and reconnecting when heartbeats are not received by vCenter Server.
  4. Perform packet captures while initiating continuous pings to the default gateway of the ESXI host in DCUI and check the drop point of packets in the network.
    Refer below kb article for performing packet captures:- Packet capture on ESXi using the pktcap-uw tool
  5. Based on the packet captures performed in point 4 if the packets are dropping in the physical network, to resolve the issue engage physical network team/administrator.
  6. Verify that network connectivity exists from vCenter Server to the ESXi host with the IP and FQDN. For more information, see Testing network connectivity with the ping command.
  7. Verify connectivity from vCenter Server to/from the ESXi host on TCP/UDP port 902 and 443. For more information, see Testing port connectivity with Telnet.
  8. Verify if restarting the ESXi Management Agents resolves the issue. For more information, see Restarting the Management agents on an ESXi host.
  9. Verify if the hostd process has stopped responding on the affected ESXi host. For more information, see Troubleshooting vmware-hostd service if it fails or stops responding on an ESX/ESXi host  
  10. Verify if the ESXi host has experienced a Purple Diagnostic Screen. For more information, see Interpreting an ESX/ESXi host purple diagnostic screen
  11. ESXi hosts can disconnect from vCenter Server due to underlying storage issues. For more information, see Identifying Fibre Channel, iSCSI, and NFS storage issues on ESXi/ESX hosts.
  12. ESXi host "Cannot contact the specified host hostname\IP. For more information, see The host appears as disconnected in vCenter.
  13. Verify the LACP (lag) configuration in vCenter for the affected ESXi host. For more information, see LACP configuration

Additional Information

Powered by Wolken Software