Getting error 'LDAP exception error code 49 (invalidCredentials)' when executing vCert script as part of renewing the Machine SSL certificate to fix the 'no healthy upstream error'.

Products

• VMware vCenter Server

Issue/Introduction

• The following error is noticed when trying to access the vCenter Server
  
  'no healthy upstream' screen'
  
  
• Services fails to start if we perform restart of all vCenter services: 
  service-control --stop --all && service-control --start --all 

• Expired Machine SSL certificate found when checking the certificate validity by executing following command: 
  for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;
  
  
• The certificate replacement tool 'vCert' fails to execute with following error: 
  
  python vCert.py
  
  'lib.exceptions.LdapException: LDAP exception error code 49 (invalidCredentials)'
  
  

Environment

VMware vCenter server 8.x

Cause

• Expired machine SSL certificate present in certificate store of vCenter server causing the service start failure. 
• The execution of vCert script fails due to machine account password stored in the VMDIRD database does not match the password used by the services to authenticate with the VMware Directory Service.
• Both the credentials should match in order to perform successful execution of the vCert script. 

Resolution

Important note: Ensure availability of vCenter server snapshot before attempting to execute the script file. If the vCenter Server is in linked mode with other vCenter Servers, refer to the article below for taking offline snapshots of each linked vCenter Server node. VMware vCenter in Enhanced Linked Mode pre-changes snapshot (online or offline) best practice

• Follow the steps to execute the 'reset_machine_pw.sh' script as outlined in this KB article to resolve this error 'LDAP exception error code 49 (invalid Credentials)' KB316608
• The vCert.py script should be executable to replace the expired machine SSL certificate. 
  Reference: vCert - Scripted vCenter expired certificate replacement