vCenter upgrade fail | Encountered an internal error. see /var/log/firstboot/vmafd-firstboot.py_#####_stderr.log

Products

• VMware vCenter Server

Issue/Introduction

• Upgrade from vCenter 8 to 9 fails at stage 2

• In the  /var/log/firstboot/vmafd-firstboot.py_ ####_stderr.log
  
  

  YYYY-MM-DDTHH:MM:SS  password:
YYYY-MM-DDTHH:MM:SS  Container ou=Computers,dc=domain,dc=local already exists, not added.
Container cn=Certificate-Authorities,cn=Configuration,dc=domain,dc=local already exists, not added.
Group cn=DCClients,cn=Builtin,dc=domain,dc=local already exists, not added.
Group cn=CAAdmins,cn=Builtin,dc=domain,dc=local already exists, not added.
Set dcAccount registry key to vcenter.domain.local
Failed to UpdateDCActSRPSecret (9234)
Vdcupgrade failed. Error[9234] - User invalid credential

  YYYY-MM-DDTHH:MM:SS  <class 'SystemExit'>
YYYY-MM-DDTHH:MM:SS  <class 'SystemExit'>


• In the /var/log/vmware/vmdird/vmdird.log

ERROR: SASLSessionStep: sasl error (-13)(SASL(-13): authentication failure: client evidence does not match what we calculated. Probably a password error)
ERROR: VdirPasswordFailEvent from user(cn=vcenter.domain.local,ou=domain controllers,dc=domain,dc=local), error(0)()
ERROR: VmDirSendLdapResult: Request (Bind), Error (LDAP_INVALID_CREDENTIALS(49)), Message ((49)(SASL step failed.)), (0) socket (127.0.0.1)
ERROR: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "cn=vcenter.domain.local,ou=Domain Controllers,dc=domain,dc=local", Method: SASL

Environment

vCenter 8.x

vCenter 9.x

Cause

This issue occurs when a machine loses trust because the account in vmdird has a password mismatch, as indicated in vmdird-syslog.log.

This typically happens after restoring the vCenter Server from an older backup or snapshot.

Resolution

Reset using the reset_machine_pw.sh script (Built in)

1. Take offline snapshots of all vCenters in the SSO domain before proceeding. 
   
   
2. Connect to the vCenter over SSH with the root user and type shell to access the bash shell
   
   
3. Run the script using the command below - the prompt will require the FQDN of the replication partners (vCenters) where the machine account password needs to be reset, and also prompt for SSO admin credentials:
   
   # /usr/lib/vmware-vmdir/vmdir-tool/reset_machine_pw.sh

Additional Information

LDAP Error Code 49"/Error (49) error in vmdird logs in vCenter Server