"Failed to init PSC and/or Postgres. Check the SDDC Manager UI Server logs for more details." VCF UI is not accessible.

Products

• VMware SDDC Manager

Issue/Introduction

• VCF UI becomes inaccessible with an error "Failed to init PSC and/or Postgres. Check the SDDC Manager UI Server logs for more details."

• SDDC Manager UI fails to load with error - "VMware Cloud Foundation is initializing"

• Following logs are observed in /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log

yyyy-mm-ddThh:mm:ss.mss+0000 VERBOSE [a4c53bab982c432c, 3878912b2d1849e4, d7705f9eba5a4aa5] [services/logging/opentrace.js, init-pscs-primary, _finish:231] Span init-pscs-primary complete: Total time: 2511ms {"duration":2511}
yyyy-mm-ddThh:mm:ss.mss+0000 ERROR [a4c53bab982c432c, 3878912b2d1849e4] [services/pscUtils.js, init-pscs, attemptPSCInit:67] Caught error from await primaryPscInit
yyyy-mm-ddThh:mm:ss.mss+0000 WARN [a4c53bab982c432c, 3878912b2d1849e4] [services/pscUtils.js, init-pscs, attemptPSCInitWithRetry:111]
100.109: VError: PSC Initilization attempt "6" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
    at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
    at attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:104:46)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Error Info: {"retryCount":6,"status":403,"errorModule":100,"errorCode":109,"code":"ERR_BAD_RESPONSE"}
caused by:
100.108: VError: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
    at Object.initiatePscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
    at attemptPSCInit (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:72:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:99:17)
Error Info: {"errorModule":100,"errorCode":108,"code":"ERR_BAD_RESPONSE"}

Environment

VCF 5.x

Cause

• On boot, the SDDC tries to establish an SSH connection to the Management Domain vCenter.

• SDDC Manager failed to access Management Domain vCenter Server SSH, which may be caused by the reasons below:

  • SSH is deactivated on Management Domain vCenter Server.

  • The root password on the Management Domain vCenter has been expired.

  • The root password has been changed on the Management Domain vCenter, the UI will not launch correctly as SDDC Manager is unaware of the new root password on the vCenter side and will continue to try to connect using the old password.

Resolution

1. Activate SSH on Management Domain vCenter Server following: Enable or Disable SSH and Bash Shell Access
2. Restart the sddc-manager-ui-app    

   systemctl restart sddc-manager-ui-app

3. If step 2 doesn't work, please change the root password on the Management domain vCenter and update the SDDC Manager with those new credentials following the steps below. 

• Generate a TOKEN on the SDDC Manager as root.

  TOKEN=$(curl -d '{"username" : "<sso username>", "password" : "<sso password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')

• Pull the vCenter root credentials.

  curl -X GET 'localhost/v1/system/credentials?entityType=VCENTER&credentialType=SSH' -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN"

   
• Put the new credentials into the SDDC DB.

  curl -X PUT 'localhost/v1/system/credentials/<id from above step>' -d '<new password>' -H "Content-type:application/json" -H "Authorization: Bearer $TOKEN"| json_pp

• Restart the sddc-manager-ui-app    

  systemctl restart sddc-manager-ui-app

Additional Information

• The error "Failed to initiate PSC: Primary psc init failed and failover psc init also failed" will also be logged if the vCenter server SSH Host Keys stored in SDDC manager contain a mismatch.
• Run the script from KB - How to update the SSH host keys on the SDDC Manager, to update the host keys of vCenter in SDDC.