"Failed to init PSC and/or Postgres. Check the SDDC Manager UI Server logs for more details." VCF UI is not accessible.
search cancel

"Failed to init PSC and/or Postgres. Check the SDDC Manager UI Server logs for more details." VCF UI is not accessible.

book

Article ID: 389716

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • VCF UI becomes inaccessible with an error "Failed to init PSC and/or Postgres. Check the SDDC Manager UI Server logs for more details."

 

  • SDDC Manager UI fails to load with error - "VMware Cloud Foundation is initializing"

  • Following logs are observed in /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log

    YYYY-MM-DDTHH:MM:SS.mss+0000 VERBOSE [a4c53bab982c432c, 3878912b2d1849e4, d7705f9eba5a4aa5] [services/logging/opentrace.js, init-pscs-primary, _finish:231] Span init-pscs-primary complete: Total time: 2511ms {"duration":2511}
YYYY-MM-DDTHH:MM:SS.mss+0000 ERROR [a4c53bab982c432c, 3878912b2d1849e4] [services/pscUtils.js, init-pscs, attemptPSCInit:67] Caught error from await primaryPscInit
YYYY-MM-DDTHH:MM:SS.mss+0000 WARN [a4c53bab982c432c, 3878912b2d1849e4] [services/pscUtils.js, init-pscs, attemptPSCInitWithRetry:111]
100.109: VError: PSC Initilization attempt "6" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
    at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
    at attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:104:46)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Error Info: {"retryCount":6,"status":403,"errorModule":100,"errorCode":109,"code":"ERR_BAD_RESPONSE"}
caused by:
100.108: VError: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
    at Object.initiatePscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
    at attemptPSCInit (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:72:26)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:99:17)
Error Info: {"errorModule":100,"errorCode":108,"code":"ERR_BAD_RESPONSE"}


YYYY-MM-DDTHH:MM:SS.mss+0000 ERROR [c2f2b47783674dbc, f81e250ca5e64fc2, 3e9563ebd619463c] [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:58] axios.error.response.data {"data":"Error executing remote command via SSH: WARNING: Your password has expired.\nPassword change required but no TTY available."}


YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]:                 timeout = setTimeout(reject, sshCommandTimeout, new Error(failureMessage));
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]:                                                                 ^
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]: Error: Remote ssh command timed out
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]:     at /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275:65
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]:     at new Promise (<anonymous>)
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]:     at init (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:274:24)
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]:     at Object.reset (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:294:13)
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]:     at initializeSSO (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:60:24)
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]: Node.js v18.20.4
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]:                timeout = setTimeout(reject, sshCommandTimeout, new Error(failureMessage));
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]:                                                                ^
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]: Error: Remote ssh command timed out
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]:    at /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275:65
YYYY-MM-DDTHH:MM:SS.mss+0000  ap-sy5-vcf sddc-manager-ui-app[70645]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275

     

  • Due to this issue, VCF Fleet management may not be able to retrieve tasks with an error  "Retrieving task counts failed. Http failure response for https://<Ops_FQDN>/vcf-operations/plug/sddc-###/plugin/api/v1/tasks?pageSize=1: 503 OK"

Environment

  • VCF 5.x
  • VCF 9.X

Cause

  • On boot, the SDDC tries to establish an SSH connection to the Management Domain vCenter.

  • SDDC Manager failed to access Management Domain vCenter Server SSH, which may be caused by the reasons below:

    • SSH is deactivated on Management Domain vCenter Server.

    • The root password on the Management Domain vCenter has been expired.

    • The root password has been changed on the Management Domain vCenter, the UI will not launch correctly as SDDC Manager is unaware of the new root password on the vCenter side and will continue to try to connect using the old password.

Resolution

  1. Activate SSH on Management Domain vCenter Server following: Enable or Disable SSH and Bash Shell Access
  2. Restart the sddc-manager-ui-app    
    systemctl restart sddc-manager-ui-app
  3. If step 2 doesn't work, please change the root password on the Management domain vCenter and update the SDDC Manager with those new credentials following the steps below. 
  • Generate a TOKEN on the SDDC Manager as root
    TOKEN=$(curl -d '{"username" : "<sso username>", "password" : "<sso password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')
  • Pull the vCenter root credentials.
    curl -X GET 'localhost/v1/system/credentials?entityType=VCENTER&credentialType=SSH' -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN"
     
  • Put the new credentials into the SDDC DB.
    curl -X PUT 'localhost/v1/system/credentials/<id from above step>' -d '<new password>' -H "Content-type:application/json" -H "Authorization: Bearer $TOKEN"| json_pp
  • Restart the sddc-manager-ui-app
    systemctl restart sddc-manager-ui-app
  • If the SDDC Manager UI remains inaccessible after completing the preceding troubleshooting steps, perform a full restart of the vCenter Server management services, clear your browser cache, and attempt to re-establish the session in a new window.

Additional Information

  • The error "Failed to initiate PSC: Primary psc init failed and failover psc init also failed" will also be logged if the vCenter server SSH Host Keys stored in SDDC manager contain a mismatch.
  • Run the script from KB - How to update the SSH host keys on the SDDC Manager, to update the host keys of vCenter in SDDC.
Powered by Wolken Software