Error: Failed to init PSC and/or Postgres - SDDC Manager UI is inaccessible

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

SDDC Manager UI becomes inaccessible with an error message: Failed to init PSC and/or Postgres. Check the SDDC Manager UI Server logs for more details.
The SDDC Manager UI is inaccessible and hangs on a loading screen displaying: "VMware Cloud Foundation is initializing"

The following log events were captured within the SDDC Manager at /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log:

YYYY-MM-DDT HH:MM:SS VERBOSE [a4c53bab982c432c, 3878912b2d1849e4, d7705f9eba5a4aa5] [services/logging/opentrace.js, init-pscs-primary, _finish:231] Span init-pscs-primary complete: Total time: 2511ms {"duration":2511}
100.109: VError: PSC Initialization attempt "6" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)

Error Info: {"retryCount":6,"status":403,"errorModule":100,"errorCode":109,"code":"ERR_BAD_RESPONSE"}
caused by:
100.108: VError: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
at Object.initiatePscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
Error Info: {"errorModule":100,"errorCode":108,"code":"ERR_BAD_RESPONSE"}
YYYY-MM-DDT HH:MM:SS ERROR [c2f2b47783674dbc, f81e250ca5e64fc2, 3e9563ebd619463c] [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:58] axios.error.response.data {"data":"Error executing remote command via SSH: WARNING: Your password has expired.\nPassword change required but no TTY available."}
Or, the following log events were captured within the SDDC Manager at /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log:

YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275
YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: timeout = setTimeout(reject, sshCommandTimeout, new Error(failureMessage));
YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: Error: Remote ssh command timed out
YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: at /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275:65
YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: at new Promise (<anonymous>)
YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: Node.js v18.20.4
YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275 ^
YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: Error: Remote ssh command timed out
When this issue occurs, VCF Fleet management might fail to fetch tasks and present the following error:
Retrieving task counts failed. Http failure response for https://<Ops_FQDN>/vcf-operations/plug/sddc-###/plugin/api/v1/tasks?pageSize=1: 503 OK
In the VCFOPS UI, the SDDC Manager tile (under Administration) only loads a blank white screen.

Environment

VMware Cloud Foundation 5.x
VMware Cloud Foundation 9.x

Cause

This issue occurs if the SDDC Manager fails to establish an SSH connection with the Management Domain vCenter during bootup. Common causes include:

Deactivated SSH on the Management Domain vCenter Server.
Expired root password on the Management Domain vCenter.
The root password was changed natively on the vCenter without updating the credentials in the SDDC Manager database.

Resolution

Enable SSH on the Management Domain vCenter Server Enable or Disable SSH and Bash Shell Access
Restart the sddc-manager-ui-app service: systemctl restart sddc-manager-ui-app
If Step 2 does not resolve the issue, change the root password on the Management Domain vCenter and update the SDDC Manager with the new credentials using the steps below.
1. Log in to the SDDC Manager as root and generate an API token:
  # TOKEN=$(curl -d '{"username" : "<sso username>", "password" : "<sso password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')
2. Use the generated token to query the system for the vCenter root credentials and locate its ID:
  # curl -X GET 'localhost/v1/system/credentials?entityType=VCENTER&credentialType=SSH' -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN"
3. Push the new password into the SDDC Database (replace `<id from above step>` and `<new password>` with your actual values):
  # curl -X PUT 'localhost/v1/system/credentials/<id from above step>' -d '<new password>' -H "Content-type:application/json" -H "Authorization: Bearer $TOKEN"| json_pp
4. Restart the sddc-manager-ui-app service:
  # systemctl restart sddc-manager-ui-app
5. If the SDDC Manager UI remains inaccessible after completing the above steps, execute a full restart of the vCenter Server management services, clear the web browser's cache, and attempt to re-establish the connection in a new or incognito window.

Additional Information

The error "Failed to initiate PSC: Primary psc init failed and failover psc init also failed" will also be logged if the vCenter server SSH Host Keys stored in SDDC manager contain a mismatch.

To resolve this mismatch, run the script from the article How to update the SSH host keys on the SDDC Manager to update the host keys of vCenter in SDDC.

See KB, In SDDC Manager, vCenter Upgrade Precheck fails with the error: "vCenter root account is locked or has expired" if ED25519 has been enabled on the vCenter server.