vpxuser password change fails repeatedly

Products

• VMware vSphere ESXi

Issue/Introduction

 

• When vCenter Server attempts to change the vpxuser password on an ESXi host, the task fails.
  
  
  
• In /var/log/vmware/vpxd/vpxd.log on vCenter Server:
  
  
  The first attempt fails with a SystemError, indicating it could not save to persistent storage due to a validation failure:
  
  [timestamp] info vpxd[18618] [Originator@6876 sub=vmomi.soapStub[749] opID=HeartbeatStartHandler-77efda80-49685b09] SOAP request returned HTTP failure; <SSL(<io_obj p:0x00
007ffb5c5a90b0, h:68, <TCP '<vCenter IP> : 44682'>, <TCP '<ESXi IP> : 443'>>), /sdk>, method: updateUser; code: 500(Internal Server Error); fault: (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = (vmodl.LocalizableMessage) [
-->       (vmodl.LocalizableMessage) {
-->          key = "com.vmware.vim.host.LocalAccountManager.configStoreError",
-->          arg = <unset>,
-->          message = "Error while storing user account information to persistent storage. The new information may not be available after reboot."
-->       }
-->    ],
-->    reason = "Failed to save to persistent storage: Failed to validate Set"
-->    msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00007ffb5c5a90b0, h:68, <TCP '<vCenter IP> : 44682'>, <TCP '<ESXi IP> : 443'>>), /sdk>]: updateUser
--> A general system error occurred: Failed to save to persistent storage: Failed to validate Set"
--> }
[timestamp] error vpxd[18618] [Originator@6876 sub=InvtHost opID=HeartbeatStartHandler-77efda80-49685b09] Failed to change password on host esxi13.gsslabs.org: N5Vmomi5Fau
lt11SystemError9ExceptionE(Fault cause: vmodl.fault.SystemError
--> )
  
  
  Subsequent attempts fail with an InvalidLogin error, as the password is now out of sync:
  
  [timestamp] error vpxd[18730] [Originator@6876 sub=InvtHost opID=HeartbeatStartHandler-77efda80-e4fda64] Failed to change password on host <ESXi Host Name>: N3Vim5Fault12InvalidLogin9ExceptionE(Fault cause: vim.fault.InvalidLogin
--> )
  
  
  
  
• In /var/run/log/hostd.log on the ESXi host:
  
  [timestamp] In(166) Hostd[132216]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=HeartbeatStartHandler-77efda80-49685b09-d00e sid=529ec3ee user=vpxuser] Event 342 : Password was changed for account vpxuser on host <ESXi Host Name>
[timestamp] In(166) Hostd[132216]: [Originator@6876 sub=Libs opID=HeartbeatStartHandler-77efda80-49685b09-d00e sid=529ec3ee user=vpxuser] info [ConfigStore:ee9c240700] [cs:1:232624312]BeginTransaction invoked.
[timestamp] In(166) Hostd[132216]: [Originator@6876 sub=Libs opID=HeartbeatStartHandler-77efda80-49685b09-d00e sid=529ec3ee user=vpxuser] info [ConfigStore:ee9c240700] [cs:1:232624312]Transaction started, level = 1
[timestamp] In(166) Hostd[132216]: [Originator@6876 sub=Libs opID=HeartbeatStartHandler-77efda80-49685b09-d00e sid=529ec3ee user=vpxuser] info [ConfigStore:ee9c240700] Checking for empty objects and arrays in comp esx grp authentication key user_accounts id <username> object
[timestamp] Er(163) Hostd[132216]: [Originator@6876 sub=Libs opID=HeartbeatStartHandler-77efda80-49685b09-d00e sid=529ec3ee user=vpxuser] error [ConfigStore:ee9c240700] [1083]Validation Error: '/password_hash' String does not match pattern: ^[$][^:|
[timestamp] Er(163) Hostd[132216]: [Originator@6876 sub=Libs opID=HeartbeatStartHandler-77efda80-49685b09-d00e sid=529ec3ee user=vpxuser] ]+
[timestamp] Er(163) Hostd[132216]: [Originator@6876 sub=Libs opID=HeartbeatStartHandler-77efda80-49685b09-d00e sid=529ec3ee user=vpxuser] error [ConfigStore:ee9c240700] [1089] Failed to validate Set
::
[timestamp] In(166) Hostd[132236]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 362 : Multiple remote login failures detected for ESXi local user account 'vpxuser'.
  
  
  
  

• In /etc/shadow on the ESXi host:

  The password hash field in the /etc/shadow file for one or more users contains an unrecognized or invalid format, such as '!!' , instead of a valid hash.
  
  <username>:!!:xxxxxx:0:99999:7:::
  
  

 

 

 

Cause

The ESXi host's configstore does not support the !! (double exclamation mark) entry in the password hash field of the /etc/shadow file.

This causes a validation error when the file is parsed, even if the invalid entry is not for the vpxuser itself.

Resolution

To resolve this issue, replace the invalid !! entry with a character recognized by the ESXi configstore.

If the intention is to disable the account password, use a single * (asterisk) or a single ! (exclamation mark) instead.
These characters are correctly recognized by the configstore as indicating a disabled password.