The vpxd service crashes when its unable to connect to ESXi hosts with vpxuser credentials

Products

• VMware vCenter Server

Issue/Introduction

• The vpxd service crashes creating a core.vpxd-worker file in /var/core
• Messages similar to the following in vpxd.log indicate that host syncs are constantly failing due to an invalid login.

warning vpxd[28955] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-12345-48c4r5v9] Exception occurred during host sync; Got method fault; [vim.HostSystem:host-12345,esxi01.example.com], e: N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin

• The host mentioned in the error above has corresponding messages in the vpxa logs indicating incorrect vpxuser credentials

[YYYY-MM-DDTHH:MM:SS] info vpxa[2104542] [Originator@6876 sub=Default opID=HostSync-host-12345-48c4r5v9-d6] [VpxLRO] -- ERROR lro-1126689 -- vpxa -- vpxapi.VpxaService.login: vim.fault.InvalidLogin:
--> Result:
--> (vim.fault.InvalidLogin) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<cs p:0000006278a33660, TCP:localhost:443>]: login
--> Cannot complete login due to an incorrect user name or password."
--> }
--> Args:
-->
--> Arg userName:
--> "vpxuser"

 

• Some hosts in the vCenter inventory may show "null, null, null" in the hypervisor field of the Summary tab in the vSphere Client

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 7.0.x

Cause

The vpxuser credentials for each ESXi host are refreshed at a normal interval. Hashes of these passwords are also stored within vCenter. When vCenter is reverted to a snapshot, it can cause some of the rotating vpxuser passwords that it stores to also be reverted to a previous version. vCenter is no longer able to contact the ESXi hosts with these credentials and will occasionally encounter a race condition that results in the vpxd service crashing and creating a coredump.

Resolution

This issue is resolved in VMware vCenter Server 7.0 Update 3e (build number 19717403).

Please note that the workaround steps below are still necessary to correct the vpxuser password, which causes other issues such as HA agent deployment failures.

 

Workaround:

To work around this issue, you must disconnect and reconnect any hosts that vCenter is no longer able to login to with vpxuser. Reconnecting the hosts will require entering root credentials.

1. To generate a list of affected hosts, connect to the vCenter Appliance via SSH and run the following command:

journalctl -b | grep "HostSyncFailedEvent.*Cannot complete login due to an incorrect" | awk '{print $(NF-11)}' | sort | uniq

2. From the vSphere Client, disconnect and reconnect each of the hosts listed. The reconnect will initially fail with an invalid credential error, where you are then guided to re-entering in the host's root credentials.