Hosts disconnected from vCenter with error: vCenter cannot synchronize hosts

Products

• VMware vSphere ESXi

Issue/Introduction

• ESXi host(s) in vCenter shows either Disconnected or not responding.
• Reconnecting them with the vCenter fails with error: incorrect username or password.
• Accessing the ESXi host(s) via GUI or SSH also fails sometime.
• From the SSH, vpxa and hostd services are in running state.
• Host is responsive and the same password works in UI and putty session but it would not connect to vCenter server.
• From the affected host(s)' /var/run/log/hostd.log, below snippets are available:

  YYYY-MM-DDTHH:MM:SS YYYY-MM-DDTHH:MM:SS Hostd[2470198]: [Originator@6876 sub=Solo.Vmomi opID=#####-####-auto-1wdd-h5:#####-a3-4972 sid=##### user=root:VSPHERE.LOCAL\Administrator] Arg user:
YYYY-MM-DDTHH:MM:SS Hostd[####]: --> (vim.host.LocalAccountManager.AccountSpecification) {
YYYY-MM-DDTHH:MM:SS Hostd[####]: -->    id = "vpxuser",
YYYY-MM-DDTHH:MM:SS Hostd[####]: -->    password = (not shown),
YYYY-MM-DDTHH:MM:SS Hostd[####]: -->    description = "VMware Workstation administration account"
YYYY-MM-DDTHH:MM:SS Hostd[####]: --> }
YYYY-MM-DDTHH:MM:SS Hostd[2470198]: [Originator@6876 sub=Solo.Vmomi opID=#####-####-auto-1wdd-h5:#####-a3-4972 sid=##### user=root:VSPHERE.LOCAL\Administrator] Throw vmodl.fault.SystemError
YYYY-MM-DDTHH:MM:SS Hostd[2470198]: [Originator@6876 sub=Solo.Vmomi opID=#####-####-auto-1wdd-h5:#####-a3-4972 sid=##### user=root:VSPHERE.LOCAL\Administrator] Result:
YYYY-MM-DDTHH:MM:SS Hostd[####]: --> (vmodl.fault.SystemError) {
YYYY-MM-DDTHH:MM:SS Hostd[####]: -->    faultMessage = (vmodl.LocalizableMessage) [
YYYY-MM-DDTHH:MM:SS Hostd[####]: -->       (vmodl.LocalizableMessage) {
YYYY-MM-DDTHH:MM:SS Hostd[####]: -->          key = "com.vmware.vim.host.LocalAccountManager.configStoreError",
YYYY-MM-DDTHH:MM:SS Hostd[####]: -->       }
  
  

• From the vCenter server's /var/log/vmware/vpxd/vpxd.log, below log snippets are available:

  vpxd [Originator@6876 sub=vmomi.soapStub[####] opID=WorkQueue-2ea7a36b] SOAP request returned HTTP failure; <<io_obj p:0x00007fbcd413e308, h:16, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#####/sdk>, method: login; code: 500(Internal Server Error); fault: (vim.fault.InvalidLogin) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007fbcd413e308, h:16, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-#####/sdk>]: login
--> Cannot complete login due to an incorrect user name or password."

  error vpxd [Originator@6876 sub=Vmomi opID=WorkQueue-2ea7a36b] Failed to login on host; <</hgw/host-#####>, /sdk>, N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin

Cause

vCenter server is unable to read the root password stored under configstore of the host.

Resolution

Change the root password of the affected host(s) and reconnect it to the vCenter server.