Unable to Add ESXi Host to vCenter After Removal – "A general system error occurred: Failed to save to persistent storage: Failed to validate Set"

Products

• VMware vCenter Server

Issue/Introduction

When attempting to add an ESXi host back to vCenter Server after it was removed, the operation fails with a system error similar to the following in the vCenter's /var/log/vmware/vpxd/vpxd.log:

SOAP request returned HTTP failure; <SSL(<io_obj p:0x00007f2f3c8493b0, h:40, <TCP '<vCenter IP : 60204'>, <TCP '<ESXi_Host_IP : 443'>>), /sdk>, method: updateUser; code: 500(Internal Server Error)
fault: (vmodl.fault.SystemError) {
   faultMessage = (vmodl.LocalizableMessage) [
      message = "Error while storing user account information to persistent storage. The new information may not be available after reboot."
   ],
   reason = "Failed to save to persistent storage: Failed to validate Set"
}

Additionally, hostd.log on the ESXi host records validation errors similar to:

error [ConfigStore:9936c15700] [1083] Validation Error: '/password_hash' String does not match pattern: ^[$][^:|]+
error [ConfigStore:9936c15700] [1089] Failed to validate Set

Environment

VMware vSphere 7.x

VMware vSphere 8.x

VMware vSphere 9.x

Cause

The issue occurs because the ESXi user account configuration includes a password hash that starts with a “!” character.

For example:

<example_username>:!$6$e#######.9QrTH9t$5hhg#########AmgPmYSeeafoohrH7#######DQlLKkUnazL2BF#########dArUDA/Tp.:####1:0:365:7

The “!” prefix indicates that the account is locked.
When vCenter attempts to validate or update local accounts (such as creating the vpxuser), the config store validation fails due to the unsupported password pattern, resulting in:

“Error while storing user account information to persistent storage.”

Resolution

Remove the affected user account from the ESXi host:

esxcli system account remove -i <example_username>

Verify that the user is no longer present in the following files:

cat /etc/passwd
cat /etc/shadow

Confirm that the user entry (e.g., example_username) has been removed.

Reattempt to add the host back to the vCenter Server.

After the invalid user entry is removed, the host should register successfully.

 

 

Additional Information

Note: Account locking is not supported in ESXi.

Reference: ESXi Account Lockout Behavior

 

Note: You may need to remove or modify the entry in /etc/passwd If the esxcli system account remove command returns "Access to perform the operation was denied"

Reference: vpxuser password change fails repeatedly