vmware-stsd and vmware-vpxd services fail to start with ERROR_LOGON_FAILURE

Products

• VMware vCenter Server

Issue/Introduction

• Unable to access vSphere Web Client of vCenter Server with the error message 'no healthy upstream'.
• SSH vCenter Server and check the services status with command 'service-control --status' that services vmware-stsd and vmware-vpxd are stopped.
• In the /var/log/vmware/vmon/vmon.log of vCenter Server:
  
  [YYYY-MM-DDTHH:MM:SS].814Z Wa(03) 3527372 <sts> Service pre-start command's stderr: Traceback (most recent call last):
[YYYY-MM-DDTHH:MM:SS].814Z Wa(03) 3527372   File "/usr/lib/vmidentity/install/sts-prestart-script.py", line 555, in <module>
[YYYY-MM-DDTHH:MM:SS].815Z Wa(03) 3527372     raise e
[YYYY-MM-DDTHH:MM:SS].815Z Wa(03) 3527372   File "/usr/lib/vmidentity/install/sts-prestart-script.py", line 551, in <module>
[YYYY-MM-DDTHH:MM:SS].815Z Wa(03) 3527372     sts_prestart_setup_service_account()
[YYYY-MM-DDTHH:MM:SS].815Z Wa(03) 3527372   File "/usr/lib/vmidentity/install/sts-prestart-script.py", line 164, in sts_prestart_setup_service_account
[YYYY-MM-DDTHH:MM:SS].815Z Wa(03) 3527372     _create_sso_group("ActAsUsers", "Act-As Users")
[YYYY-MM-DDTHH:MM:SS].815Z Wa(03) 3527372   File "/usr/lib/vmidentity/install/sts-prestart-script.py", line 137, in _create_sso_group
[YYYY-MM-DDTHH:MM:SS].815Z Wa(03) 3527372     if sso_group_svc.group_exists(group_name) == True:
[YYYY-MM-DDTHH:MM:SS].815Z Wa(03) 3527372   File "/usr/lib/vmware/site-packages/cis/vecs.py", line 374, in group_exists
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372     raise InvokeCommandException(error)
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372 <sts> Service pre-start command's stderr: cis.exceptions.InvokeCommandException: {
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372     "detail": [
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372         {
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372             "id": "install.ciscommon.command.errinvoke",
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372             "translatable": "An error occurred while invoking external command : '%(0)s'",
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372             "args": [
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372                 "Error 46 while finding SSO group \"ActAsUsers\":\ndir-cli failed. Error 1326: Operation failed with error ERROR_LOGON_FAILURE (1326) \n"
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372         ],
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372         "localized": "An error occurred while invoking external command : 'Error 46 while finding SSO group \"ActAsUsers\":\ndir-cli failed. Error 1326: Operation failed with error ERROR_LOGON_FAILURE (1326) \n"
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372       }
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372     ],
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372     "componentKey": null,
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372     "problemId": null,
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372     "resolution": null
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372 }
[YYYY-MM-DDTHH:MM:SS].816Z Wa(03) 3527372
[YYYY-MM-DDTHH:MM:SS].033Z No(00) 3527372 <<< file Throttled >>>
[YYYY-MM-DDTHH:MM:SS].033Z Er(02) 3527372 <sts> Service pre-start command failed with exit code 1.

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Cause

vCenter Server machine account password is expired that cause sts unable to authenticate with the vmdir. vmware-vpxd depends on the vmware-stsd which must be running to let vmware-vpxd to be running successfully. The dependency of service vmware-vpxd can be retrieved with the command:

# service-control --list-dependencies vmware-vpxd
lookupsvc
sts
vmware-vpostgres

Resolution

To confirm if the vCenter Server machine account password expired or not, check the /var/log/vmware/vmdird/vmdird.log if contains the below similar messages:

[YYYY-MM-DDTHH:MM:SS].941Z:t@140003208308288:ERROR: SASLSessionStep: sasl error (-13) (SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
[YYYY-MM-DDTHH:MM:SS].941Z:t@140003208308288:ERROR: VdirPasswordFailEvent from user(cn=<VC-FQDN>,ou=domain controllers,dc=vsphere,dc=local), error(0)( )
[YYYY-MM-DDTHH:MM:SS].942Z:t@140003208308288:ERROR: VmDirSendLdapResult: Request (Bind), Error (LDAP_INVALID_CREDENTIALS(49)), Message ((49) (SASL step failed.)), (0) socket (127.0.0.1)
[YYYY-MM-DDTHH:MM:SS].942Z:t@140003208308288:ERROR: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "cn=<VC-FQDN>,ou=Domain Controllers,dc=vsphere,dc=local", Method: SASL
[YYYY-MM-DDTHH:MM:SS].971Z:t@140003208308288:ERROR: SASLSessionStep: sasl error (-13) (SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)

To resolve the issue:

1. Take backup or snapshot for vCenter Server. If vCenter Server is Linked Mode, take offline snapshot for all vCenter Server.
2. Run the script 'reset_machine_pw.sh' to reset the vCenter Server machine account password from the article LDAP Error Code 49 : Reset Machine Account Password of vCenter Server Appliance using Shell Script.