Attempting to update vCenter Server Appliance fails with Error "VMDIR Service Is Not in 'NORMAL' State".

Products

• VMware vCenter Server

Issue/Introduction

vmdird-syslog:

[YYYY-MM-DDTHH:MM:SS] err vmdird  t@139771984733760: Failed SSL function (SSL_accept), return value (-1)
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@139771984733760: NewConnection failing with error 1
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@139771984733760: ProcessAConnection: NewConnection [32] failed with error: 1
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@139772504819264: VdirPasswordFailEvent from user(cn=<vCenter Domain Name>,ou=domain controllers,dc=vsphere,dc=local), error(0)()
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@139772504819264: Srv_rpc_srp_verifier_verify_session failed, status (382312692)
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140312328521280: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140312328521280: VdirPasswordFailEvent from user(cn=<vCenter Domain Name>,ou=domain controllers,dc=vsphere,dc=local), error(0)()
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140312328521280: VmDirSendLdapResult: Request (Bind), Error (LDAP_INVALID_CREDENTIALS(49)), Message ((49)(SASL step failed.)), (0) socket (127.0.0.1)
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140312328521280: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "cn=<vCenter Domain Name>,ou=Domain Controllers,dc=vsphere,dc=local", Method: SASL
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140311481284160: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140311481284160: VdirPasswordFailEvent from user(cn=<vCenter Domain Name>,ou=domain controllers,dc=vsphere,dc=local), error(0)()
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140311481284160: VmDirSendLdapResult: Request (Bind), Error (LDAP_INVALID_CREDENTIALS(49)), Message ((49)(SASL step failed.)), (0) socket (127.0.0.1)
[YYYY-MM-DDTHH:MM:SS] err vmdird  t@140311481284160: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "cn=<vCenter Domain Name>,ou=Domain Controllers,dc=vsphere,dc=local", Method: SASL

 

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

The VMDIR of the vCenter is confirmed to be in NORMAL state.

Note: You can verify by executing the following command on the vCenter in BASH mode:

# /usr/lib/vmware-vmafd/bin/dir-cli state get

Below is the expected output:

Directory Server State: Normal (3)

Environment

VMware vCenter Server 8.x
VMware vCenter Server 7.x

Cause

This issue occurs when a machine loses its trust due to a password mismatch in the vmdird for the account listed in the vmdird-syslog.log file.

This can occur if the vCenter Server or PSC is restored to an earlier version from backups or an older snapshot.

Resolution

Option 1:
Reset using reset_machine_pw.sh shell script 


Take offline snapshots of all vCenters in the SSO domain before proceeding. This means to power off all vCenters in the SSO domain, connect to the ESXi hosts they're placed on and snapshot each of them while in powered off state. If reverting; restore each to snapshot before powering any on. This ensure consistency of the SSO domain.
Connect to the vCenter over SSH with root user and type shell to access the bash shell
Run the script using the command below - you'll be prompted for the FQDN of the replication partners (vCenters) that you wish to reset the machine account password for and also prompted for SSO admin credentials:

/usr/lib/vmware-vmdir/vmdir-tool/reset_machine_pw.sh

Option 2:
Reset using dir-cli command (For VC7.x and VC8.x)

/usr/lib/vmware-vmafd/bin/./dir-cli computer password-reset  --login  administrator --live-dc-hostname localhost --password 'administratorSSOpassword'