vCenter cannot synchronize hosts, hosts disconnected from Vcenter server

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

hosts in Vcenter shows disconnected or not responding, with hostd and vpxa responding state (incorrect username and password)

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x.

Cause

Vcenter sever cannot read the root password stored in configstore of the host "com.vmware.vim.host.LocalAccountManager.configStoreError"

Resolution

Change the root password of the host to resolve the issue

Additional Information

Host is completely responsive and the same password works in UI and putty session but it would not connect to Vcenter server.

All services on the host is working as expected and no errors reported on the host

 

hostd snippets shows the following error

Date Db(167) Hostd[2470198]: [Originator@6876 sub=Solo.Vmomi opID=lz9x221z-88608-auto-1wdd-h5:70019337-a3-4972 sid=521566a7 user=root:VSPHERE.LOCAL\Administrator] Arg user:
Date Db(167) Hostd[2470191]: --> (vim.host.LocalAccountManager.AccountSpecification) {
Date Db(167) Hostd[2470191]: -->    id = "vpxuser",
Date Db(167) Hostd[2470191]: -->    password = (not shown),
Date Db(167) Hostd[2470191]: -->    description = "VMware Workstation administration account"
Date Db(167) Hostd[2470191]: --> }
Date In(166) Hostd[2470198]: [Originator@6876 sub=Solo.Vmomi opID=lz9x221z-88608-auto-1wdd-h5:70019337-a3-4972 sid=521566a7 user=root:VSPHERE.LOCAL\Administrator] Throw vmodl.fault.SystemError
Date In(166) Hostd[2470198]: [Originator@6876 sub=Solo.Vmomi opID=lz9x221z-88608-auto-1wdd-h5:70019337-a3-4972 sid=521566a7 user=root:VSPHERE.LOCAL\Administrator] Result:
Date In(166) Hostd[2470191]: --> (vmodl.fault.SystemError) {
Date In(166) Hostd[2470191]: -->    faultMessage = (vmodl.LocalizableMessage) [
Date In(166) Hostd[2470191]: -->       (vmodl.LocalizableMessage) {
Date In(166) Hostd[2470191]: -->          key = "com.vmware.vim.host.LocalAccountManager.configStoreError",
Date In(166) Hostd[2470191]: -->       }
Date In(166) Hostd[2470191]: -->    ],
Date In(166) Hostd[2470191]: -->    reason = "Failed to save to persistent storage: Failed to validate Set",
Date In(166) Hostd[2470191]: -->    msg = ""
Date In(166) Hostd[2470191]: --> }

Additional errors in the Vcenter logs

vpxd.log

vpxd[07320] [Originator@6876 sub=vmomi.soapStub[229082] opID=WorkQueue-2ea7a36b] SOAP request returned HTTP failure; <<io_obj p:0x00007fbcd413e308, h:16, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-1069538/sdk>, method: login; code: 500(Internal Server Error); fault: (vim.fault.InvalidLogin) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<<io_obj p:0x00007fbcd413e308, h:16, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-1069538/sdk>]: login
--> Cannot complete login due to an incorrect user name or password."

error vpxd[07320] [Originator@6876 sub=Vmomi opID=WorkQueue-2ea7a36b] Failed to login on host; <</hgw/host-1069538>, /sdk>, N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin