LDAP Error Code 49 : Reset Machine Account Password of vCenter Server Appliance using Shell Script

Products

VMware vCenter Server

Issue/Introduction

vCenter Server Replication fails due to invalid credential of machine account in VMware Directory Service (vmdird) database
- For VMware vCenter Server Appliance 7.0.x, /var/log/vmware/vmdird/vmdird-syslog.log contains the following entries:
- For VMware vCenter Server Appliance 8.0.x, /var/log/vmware/vmdird/vmdird.log contains the following entries:
```
err vmdird t@140245530842880: Bind Request Failed (x.x.x.x) error 49: Protocol version: 3, Bind DN: "cn=vcsa,ou=Domain Controllers,dc=example,dc=com", Method: SASL 
err vmdird t@140245530842880: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
```

Inventory Service in vCenter Server 6.0 / VPXD-SVCS service in vCenter Server 6.5/6.7 or 7.0 fails to start, the service log (/var/log/vmware/invsvc/inv-svc.log in vCenter Server 6.0 or /var/log/vmware/vpxd-svcs/vpxd-svcs.log in vCenter Server 6.5/6.7 or 7.0 contains the following message:

Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.cis.core.authz.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
LDAP error [code: 49]

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 7.0.x
VMware vCenter Server Appliance 8.0.x

Cause

This issue occurs when the machine account password stored in the VMDIRD database does not match the password used by the services to authenticate with the VMware Directory Service.

Resolution

NOTE: Take offline (powered off) snapshots of all the vCenter Server Instance in the same vSphere Single Sign On Domain (ELM) before attempting . This is standard best practice before making any manual changes to the VMDIRD database.

Pre-requisites

SSH access to all vCenter Server instances in the ELM environment.
administrator@<vsphere.local> credentials.
Access to the Machine Account password reset script (provided in the attachment of this KB).

Procedure to Identify and Reset Expired Machine Account Password for vCenter Server in Enhanced Linked Mode (ELM)

SSH into each vCenter Server instance that is part of ELM using root account credentials.

Identify the Instance with an Expired Machine Account Password
On each vCenter Server instance, run the following command:

ldapsearch -h localhost -p 389 -x \ -D "$(/opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\services\vmdir\]' | grep dcAccountDN | cut -d '"' -f4)" \ -w "$(/opt/likewise/bin/lwregshell list_values '[HKEY_THIS_MACHINE\services\vmdir\]' | grep dcAccountPassword | cut -d '"' -f4)" \ -b "" -s base dn

Interpret the Command Output
- If the Machine Account password is expired, output will contain:
```
ldap_bind: Invalid credentials (49)
```
- If the Machine Account password is valid, output will appear similar to:
```
# extended LDIF
# LDAPv3
dn: cn=DSE Root
# search result
search: 2
result: 0 Success
# numResponses: 2
# numEntries: 1
```
On the vCenter Server identified in Step 3 with invalid credentials, copy the reset script from this KB article (download the zip file if .sh download is blocked).

Verify vmdir Database State

Check the current state:

/usr/lib/vmware-vmafd/bin/dir-cli state get

Expected output if normal:
```
Directory Server State: Normal (3)
```

If not in Normal state, set it to Normal:

/usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL

Make the Script Executable
```
chmod +x reset_machine_pw.sh
```

Identify Replication Partner Nodes

Run the following command to list replication partners:

/usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartners -h localhost -u administrator -w <Administrator_Password>

Sample output

Partner: psc01.example.com
Partner: psc02.example.com

Run the Reset Script
```
./reset_machine_pw.sh
```
When prompted Enter the administrator@<vsphere.local> password.
When prompted Enter the FQDN(s) of replication partners (Enter a space in between each entry. Include the FQDN of the vCenter that you are running the script from. If you do not include all partners, services will not start on the node where you are running the script).

Resolve “bad interpreter” Error (If Encountered)

If running the script returns:

bash: ./reset_machine_pw.sh: /bin/bash^M: bad interpreter: No such file or directory

Run the command

sed -i -e 's/\r$//' reset_machine_pw.sh

Then re-run Step 8.
Restart All Services
```
service-control --stop --all && service-control --start --all
```
Important: Repeat Steps 8–10 on VMDIR replication partner nodes if replication is broken in both directions due to expired Machine Account credentials.

Sample Script Execution Output

Machine account password reset for vCenterext.example.com started on [YYYY-MM-DDTHH:MM:SS]

Detected this node is a vCenter server with embedded PSC.
Detected DN: cn=vCenterext.example.com,ou=Computers,dc=vsphere,dc=local
Detected PNID: vCenterext.example.com
Detected PSC: psc.example.com
Detected SSO domain name: vsphere.local

Enter password for [email protected]:
updating registry with password.
updating local PSC with password.
modifying entry "cn=vCenterext.example.com,ou=Computers,dc=vsphere,dc=local"

Since there were no replication partners specified, we're done here.
Finished on [YYYY-MM-DDTHH:MM:SS]

Additional Information

SHA256 Hashes

reset_machine_pw.zip - 0F879F459E22EC9B3917E4792F4610BA1296EBBA15BAE19667CF0EEABB53F363
reset_machine_pw.sh - D4052210BB5C1C9989C5FC4DFF7667235A12EB55F50649C972715F89237A31A8

MD5 Hashes

reset_machine_pw.zip - 32BCC25843E44E93184FFBDF5C903AA9
reset_machine_pw.sh - 32BCC25843E44E93184FFBDF5C903AA9

Attachments

reset_machine_pw.zip get_app

reset_machine_pw get_app