Error: SDDC Manager UI fails to load with "VMware Cloud Foundation is initializing" due to expired vCenter Server root password

Products

VMware Cloud Foundation

Issue/Introduction

SDDC Manager UI fails to load after reboot.
The management vCenter Server's root password has expired.
The management vCenter Server's root password was updated then vCenter was reverted to a previous snapshot.
The following errors were captured in the SDDC Manager log file located at /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log:

Error Info: {"errorModule":100,"errorCode":108,"code":"ERR_BAD_RESPONSE"}
caused by:
VError: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
at Object.primaryPscInitError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
at attemptPSCInit (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:68:30)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:99:17)
Error Info: {"errorModule":100,"errorCode":107,"code":"ERR_BAD_RESPONSE"}
caused by:

Caught ssh exec error event
WARNING: Your password has expire

VDT script reports the following error:

[FAIL] Check vCenter root credentials from SDDC Manager.
SSH attempt to the vCenter Server prompts to change the root password:
sudo: Account or password is expired, reset your password and try again
Changing password for root.
Current password:

Environment

SDDC Manager 4.x
SDDC Manager 5.x
VMware Cloud Foundation 5.x
VMware Cloud Foundation 9.x

Cause

SDDC Manager connects to the management domain vCenter Server via SSH. If the vCenter Server root password is expired, this connection attempt fails.
Updating the root password directly on the vCenter Server does not update the SDDC Manager database. Consequently, the UI fails to launch as SDDC Manager continues to authenticate using stale credentials.
Consequently, repeated failed login attempts from SDDC Manager will lock out the newly updated, valid vCenter Server root account.
This can also occur when vCenter password is updated and then snapshot is reverted. This will rollback the password on vCenter but not in SDDC Manager.

Resolution

Workaround 1

Update the vCenter Server's root password to match the password saved in the SDDC Manager database:

SSH to the SDDC Manager with vcf and then elevate to root using su.
Note down the password for the root account of the vCenter Server using the command: lookup_passwords
Reset the vCenter Sever root password. Refer Reset vCenter Server Appliance root password without reboot (6.7u1 / 7.x / 8.x).
SSH into the vCenter Server using root and clear the password history: echo "" > /etc/security/opasswd
Reset any active account lockouts for the root user: faillock --user root --reset
Set the root password to match the value found in the SDDC Manager database from step 2: passwd root
SSH to the SDDC Manager with vcf and elevate to root using su and restart the SDDC Manager services: /opt/vmware/vcf/operationsmanager/scripts/cli/sddcmanager_restart_services.sh

Workaround 2

Reset the vCenter Server password and update the SDDC Manager database with the new password:

Reset the vCenter Sever root password. Refer Reset vCenter Server Appliance root password without reboot (6.7u1 / 7.x / 8.x).
Take a snapshot of the SDDC Manager virtual machine: Taking and restoring snapshots of the SDDC Manager appliance
Update the password on the SDDC Manager. SSH to the SDDC Manager with vcf and elevate to root using su.
1. For SDDC Manager 4.x deployments, perform the following steps:
  1. Execute the following command to list the vCenter Server details from the SDDC Manager platform database and note down the id of the management vCenter Server:
    Sample output:
    vm_hostname | id | status
    ---------------------+--------------------------------------+--------
    vcsa.example.com | ########-####-####-####-########### | ACTIVE
  2. If the vCenter Server status is in 'ERROR' in the above output, set it to 'ACTIVE': psql -h localhost -U postgres -d platform -c "update vcenter set status='ACTIVE' where id='<id from 3.a.i>'";
  3. Update the SDDC Manager inventory with the new vCenter Server password created in step 1: curl -X PUT 'localhost/v1/system/credentials/<id from step 3.a.i>' -d "<new password>" -H "Content-type:application/json" -H "Authorization: Bearer $TOKEN"| json_pp
    Note: While vCenter Server supports a broad range of special characters for the root password, SDDC Manager APIs have stricter limitations. Refer to the following list when setting the vCenter Server root password:
    Accepted General characters: A-Z a-z 0-9
    Accepted Special Characters: ! @ $ % ^ & ( ) - _ + ` ~ . < > ?
    Accepted but not recommended: '
    Not Accepted Characters: # = { } [ \ ; : , / "
  4. Confirm the new password is updated in SDDC Manager: curl 'localhost/credentials-service/credentials?entityType=VCENTER&credentialType=SSH' | json_pp
2. For SDDC Manager 5.x deployments, perform the following steps:
  1. Generate a TOKEN on the SDDC Manager as root: TOKEN=$(curl -d '{"username" : "<sso username>", "password" : "<sso password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')
  2. Pull the vCenter Server root credentials: curl -X GET 'localhost/v1/system/credentials?entityType=VCENTER&credentialType=SSH' -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN"
  3. Update the SDDC Manager database with the new credentials: curl -X PUT 'localhost/v1/system/credentials/<id from step 3.b.ii>' -d "<new password>" -H "Content-type:application/json" -H "Authorization: Bearer $TOKEN"| json_pp

Additional Information

In rare instances, the SDDC Manager UI may fail to load if the default shell on the vCenter Server Appliance (VCSA) is configured as the appliance shell.

SSH to the vCenter Server as root and change to the bash shell: chsh -s /bin/bash root

Using the VCF Diagnostic Tool for vSphere (VDT)

Contact Broadcom support