"Failed to init PSC and/or Postgres. Check the SDDC Manager UI Server logs for more details." VCF UI is not accessible.
search cancel

"Failed to init PSC and/or Postgres. Check the SDDC Manager UI Server logs for more details." VCF UI is not accessible.

book

Article ID: 389716

calendar_today

Updated On:

Products

VMware SDDC Manager

Issue/Introduction

  • VCF UI becomes inaccessible with an error "Failed to init PSC and/or Postgres. Check the SDDC Manager UI Server logs for more details."

 

  • SDDC Manager UI fails to load with error - "VMware Cloud Foundation is initializing"

  • Following logs are observed in /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log
yyyy-mm-ddThh:mm:ss.mss+0000 VERBOSE [a4c53bab982c432c, 3878912b2d1849e4, d7705f9eba5a4aa5] [services/logging/opentrace.js, init-pscs-primary, _finish:231] Span init-pscs-primary complete: Total time: 2511ms {"duration":2511}
yyyy-mm-ddThh:mm:ss.mss+0000 ERROR [a4c53bab982c432c, 3878912b2d1849e4] [services/pscUtils.js, init-pscs, attemptPSCInit:67] Caught error from await primaryPscInit
yyyy-mm-ddThh:mm:ss.mss+0000 WARN [a4c53bab982c432c, 3878912b2d1849e4] [services/pscUtils.js, init-pscs, attemptPSCInitWithRetry:111]
100.109: VError: PSC Initilization attempt "6" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
    at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
    at attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:104:46)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Error Info: {"retryCount":6,"status":403,"errorModule":100,"errorCode":109,"code":"ERR_BAD_RESPONSE"}
caused by:
100.108: VError: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
    at Object.initiatePscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
    at attemptPSCInit (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:72:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:99:17)
Error Info: {"errorModule":100,"errorCode":108,"code":"ERR_BAD_RESPONSE"}


Following logs are observed in /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log

YYYY-MM-DDTHH:MM:SS:08.832634+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275
YYYY-MM-DDTHH:MM:SS:08.832979+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]:                 timeout = setTimeout(reject, sshCommandTimeout, new Error(failureMessage));
YYYY-MM-DDTHH:MM:SS:08.833032+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]:                                                                 ^
YYYY-MM-DDTHH:MM:SS:08.833078+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]: Error: Remote ssh command timed out
YYYY-MM-DDTHH:MM:SS:08.833114+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]:     at /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275:65
YYYY-MM-DDTHH:MM:SS:08.833149+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]:     at new Promise (<anonymous>)
YYYY-MM-DDTHH:MM:SS:08.833177+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]:     at init (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:274:24)
YYYY-MM-DDTHH:MM:SS:08.833205+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]:     at Object.reset (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:294:13)
YYYY-MM-DDTHH:MM:SS:08.833250+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]:     at initializeSSO (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:60:24)
YYYY-MM-DDTHH:MM:SS:08.833291+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
YYYY-MM-DDTHH:MM:SS:08.833335+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]: Node.js v18.20.4
YYYY-MM-DDTHH:MM:SS:08.832229+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275
YYYY-MM-DDTHH:MM:SS:08.832229+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]:                timeout = setTimeout(reject, sshCommandTimeout, new Error(failureMessage));
YYYY-MM-DDTHH:MM:SS:08.832229+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]:                                                                ^
YYYY-MM-DDTHH:MM:SS:08.832229+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]: Error: Remote ssh command timed out
YYYY-MM-DDTHH:MM:SS:08.832229+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]:    at /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275:65

YYYY-MM-DDTHH:MM:SS:08.832634+00:00 ap-sy5-vcf sddc-manager-ui-app[70645]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275

Due to this issue, VCF Fleet management may not be able to retrieve tasks with error "Retrieving task counts failed. Http failure response for https://<Ops_FQDN>/vcf-operations/plug/sddc-###/plugin/api/v1/tasks?pageSize=1: 503 OK"

Environment

  • VCF 5.x
  • VCF 9.X

Cause

  • On boot, the SDDC tries to establish an SSH connection to the Management Domain vCenter.

  • SDDC Manager failed to access Management Domain vCenter Server SSH, which may be caused by the reasons below:

    • SSH is deactivated on Management Domain vCenter Server.

    • The root password on the Management Domain vCenter has been expired.

    • The root password has been changed on the Management Domain vCenter, the UI will not launch correctly as SDDC Manager is unaware of the new root password on the vCenter side and will continue to try to connect using the old password.

Resolution

  1. Activate SSH on Management Domain vCenter Server following: Enable or Disable SSH and Bash Shell Access
  2. Restart the sddc-manager-ui-app     
    systemctl restart sddc-manager-ui-app
  3. If step 2 doesn't work, please change the root password on the Management domain vCenter and update the SDDC Manager with those new credentials following the steps below. 
  • Generate a TOKEN on the SDDC Manager as root 
    TOKEN=$(curl -d '{"username" : "<sso username>", "password" : "<sso password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')
  • Pull the vCenter root credentials. 
    curl -X GET 'localhost/v1/system/credentials?entityType=VCENTER&credentialType=SSH' -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN"
     
  • Put the new credentials into the SDDC DB. 
    curl -X PUT 'localhost/v1/system/credentials/<id from above step>' -d '<new password>' -H "Content-type:application/json" -H "Authorization: Bearer $TOKEN"| json_pp
  • Restart the sddc-manager-ui-app
    systemctl restart sddc-manager-ui-app
  • If the SDDC Manager UI remains inaccessible after completing the preceding troubleshooting steps, perform a full restart of the vCenter Server management services, clear your browser cache, and attempt to re-establish the session in a new window.

Additional Information

  • The error "Failed to initiate PSC: Primary psc init failed and failover psc init also failed" will also be logged if the vCenter server SSH Host Keys stored in SDDC manager contain a mismatch.
  • Run the script from KB - How to update the SSH host keys on the SDDC Manager, to update the host keys of vCenter in SDDC.
Powered by Wolken Software