LDAP Error Code 49 : Reset Machine Account Password of vCenter Server Appliance using Shell Script

Products

• VMware vCenter Server

Issue/Introduction

This KB will help to reset machine account password in VMware Directory Services Database using a shell script, it is applicable for vCenter Server Appliance only.

Symptoms:

• PSC Replication fails due to invalid credential of machine account in VMware Directory Service (vmdird) database

for VMware vCenter Server Appliance 7.0.x, /var/log/vmware/vmdird/vmdird-syslog.log contains the following entries:
for VMware vCenter Server Appliance 8.0.x, /var/log/vmware/vmdird/vmdird.log contains the following entries:

[YYYY-MM-DDTHH:MM:SS] err vmdird t@140245530842880: Bind Request Failed (x.x.x.x) error 49: Protocol version: 3, Bind DN: "cn=vcsa1,ou=Domain Controllers,dc=domain,dc=local", Method: SASL [YYYY-MM-DDTHH:MM:SS] err vmdird t@140245530842880: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)

• Inventory Service in vCenter Server 6.0 / VPXD-SVCS service in vCenter Server 6.5/6.7 or 7.0 fails to start
• the service log (/var/log/vmware/invsvc/inv-svc.log in vCenter Server 6.0 or /var/log/vmware/vpxd-svcs/vpxd-svcs.log in vCenter Server 6.5/6.7 or 7.0 contains the following message:

  Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.vmware.cis.core.authz.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
  LDAP error [code: 49]

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 7.0.x
VMware vCenter Server Appliance 8.0.x

Cause

This issue happens due to a mismatch in the machine account password stored in VMDIRD Database and the password used by services to connect to VMware Directory Service.

Resolution

NOTE: Please take offline (powered off) snapshots of all PSC's and VC's in the same vSphere Domain (or in ELM) before attempting.  This is standard best practice before making any manual changes to the PSC VMDIRD database. 

1. Copy the script attached to this article on the vCenter Server or PSC which is facing the issue with Invalid Credentials
2. verify that the vmdir database is in normal state:

   # /usr/lib/vmware-vmafd/bin/dir-cli state get
   

   The output should look like this:

   Directory Server State: Normal (3)

3. If the vmdir database is not in normal state, change it by running:

   # /usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL

4. Make the script executable by executing the following command:

   # chmod +x reset_machine_pw.sh

5. Run the script.  You will be prompted for the Administrator@<sso.domain> password and replication partner name if it is executed on PSC or Embedded Node
6. Restart all the services after modifying the password:

   # service-control --stop --all && service-control --start --all

Note: This script needs to be executed on VMDIR replication partner node as well, if replication is not working in both directions due to invalid credentials of each other's machine account.
 

Example output :

• Script executed on a Platform Services Controller with a replication partner:

a) Identify the partner nodes using vdcrepadmin command:

root@vCenter1 [ /tmp ]# /usr/lib/vmware-vmdir/bin/vdcrepadmin -f showpartnerstatus -h localhost -u administrator
password:
Partner: vCenter2.domain.local

b) Execute the script to reset password

root@vcsa1 [ /tmp ]# ./reset_machine_pw.sh
==================================
Machine account password reset for vCenter1.domain.local started on Wed Jun 19 09:09:49 UTC 2019

Detected that this node is an external PSC.
Please provide the replication partners separated by a space: vCenter2.domain.local

Detected DN: cn=vCenter1.domain.local,ou=Domain Controllers,dc=vsphere,dc=local
Detected PNID: vCenter1.domain.local
Detected PSC: vCenter1.domain.local
Detected SSO domain name: vsphere.local

Enter password for [email protected]:
updating registry with password.
updating local PSC with password.
modifying entry "cn=vCenter1.domain.local,ou=Domain Controllers,dc=vsphere,dc=local"

Updating replication partners with the new password as well.
Changing password for vCenter1.domain.local in the VMDIR database located at vCenter2.domain.local
modifying entry "cn=vCenter1.domain.local,ou=Domain Controllers,dc=vsphere,dc=local"

Finished on Wed Jun 19 09:09:57 UTC 2019

c) Execute the script on Partner Node as well if VMDIR replication is not working in both directions due to error 49. In above example, same script needs to be executed on partner node vcsa2.domain.local

• Script executed on a vCenter Server Node with an external PSC

a) Execute the script to reset the password

root@vCenterext[ /tmp ]# ./reset_machine_pw.sh
==================================
Machine account password reset for vCenterext.domain.local started on Wed Jun 19 09:19:32 UTC 2019

Detected this node is a vCenter server with external PSC.

Detected DN: cn=vCenterext.domain.local,ou=Computers,dc=vsphere,dc=local
Detected PNID: vCenterext.domain.local
Detected PSC: psc.domain.local
Detected SSO domain name: vsphere.local

Enter password for [email protected]:
updating registry with password.
updating local PSC with password.
modifying entry "cn=Centerext.domain.local,ou=Computers,dc=vsphere,dc=local"

Since there were no replication partners specified, we're done here.
Finished on Wed Jun 19 09:19:38 UTC 2019

 

Additional Information

NOTE:  You may receive an error when you try to run the script:

bash:  ./reset_machine_pw.sh: /bin/bash^M: bad interpreter: No such file or directory

This error is caused by DOS carriage returns added to the script when copying from a Windows-based text editor.  To resolve this problem:

• run the following command:

  # sed -i -e 's/\r$//' reset_machine_pw.sh

• Rerun the script.

Impact/Risks:
The article also assumes you have taken powered-off snapshots of all the vCenter Server or PSC nodes in the same vSphere Domain (ELM) prior to attempting the fix (per the instructions set forth in the resolution section of this article).

Should something go wrong, you will have to restore the snapshots taken before the attempted fix.