Host status is incorrectly stuck as CONNECTED when the host is OFFLINE

Products

• VMware vSphere ESXi

Issue/Introduction

• vCenter reports a host as CONNECTED with VMs powered on
• Host tasks via the vCenter timeout or are greyed out
• vCenter continues to report the host as ONLINE even if the host is POWERED OFF
• Host was previously unresponsive due to a different issue and has since been recovered but remains unmanageable in vCenter

Environment

6.x
7.x

Cause

vCenter logging within the vpxd.log will show the host is giving bad information

YYYY-MM-DDT00:00:00.000Z info vpxd[39496] [Originator@6876 sub=MoHost opID=SWI-########] Host [vim. HostSystem:host-###, <ESXI_HOSTNAME>]: host Config information is nullptr
YYYY-MM-DDT00:00:00.000Z error vpxd[39441] [Originator@6876 sub=licenseClient opID=LicenseClientProcessInventoryLoadedAsync-########] Bad host information for host-###
YYYY-MM-DDT00:00:00.000Z error vpxd[39441] [Originator@6876 sub=licenseClient opID=LicenseClientProcessInventoryLoadedAsync-########] UpdateAllHostsUsages Failed to get host usages,hostMoId: host-### error: N3Vim5Fault24IncorrectHostInformation9ExceptionE (Fault cause: vim. fault. IncorrectHostInformation
YYYY-MM-DDT00:00:00.000Z info vpxd[39406] [Originator@6876 sub=HostCnx opID=CheckforMissingHeartbeats-########] [VpxdHostCnx] No heartbeats received from host; cnx: ########-####-####-####-############, h: host-###, time since last heartbeat: ######ms

Resolution

1. Reboot both Host and vCenter, confirm you are not able to reconnect the host afterwards
2. If the host is still reporting as connected but unmanageable we will proceed to force disconnect the host in the vCenter database via the following command:
   
   

   /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres -c "UPDATE vpx_host SET enabled = 0 where id = <HOST_ID>;"

   NOTE: <HOST_ID> can be identified by selecting the target host in the inventory via the vCenter UI and checkign the web browser URL, host ID will be presented as host-### within the URL, the ### parameter is all that is required for the command above.

3. Restart vCenter services after:
   
   

   service-control --stop --all
   service-control --start --all

   
   
   
4. Verify within vCenter UI that host is now reporting as disconnected, proceed to reconnect and provide root credentials to recreate vpxuser service account between vCenter and ESXI
5. Verify host is now manageable