Error: Failed to init PSC and/or Postgres - SDDC Manager UI is inaccessible
search cancel

Error: Failed to init PSC and/or Postgres - SDDC Manager UI is inaccessible

book

Article ID: 389716

calendar_today

Updated On:

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

  • SDDC Manager UI becomes inaccessible with an error message: Failed to init PSC and/or Postgres. Check the SDDC Manager UI Server logs for more details.


  • The SDDC Manager UI is inaccessible and hangs on a loading screen displaying: "VMware Cloud Foundation is initializing"

  • The following log events are captured within the SDDC Manager at /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log:

    YYYY-MM-DDT HH:MM:SS VERBOSE [#############, ################, ##############] [services/logging/opentrace.js, init-pscs-primary, _finish:###] Span init-pscs-primary complete: Total time: ####ms {"duration":####}
    ###.###: VError: PSC Initialization attempt "6" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
        at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:###:#)

    Error Info: {"retryCount":6,"status":403,"errorModule":100,"errorCode":109,"code":"ERR_BAD_RESPONSE"}
    caused by:
    ###.###: VError: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
        at Object.initiatePscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:###:#)
    Error Info: {"errorModule":100,"errorCode":108,"code":"ERR_BAD_RESPONSE"}
    YYYY-MM-DDT HH:MM:SS ERROR [##############, ############, ###############] [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:##] axios.error.response.data {"data":"Error executing remote command via SSH: WARNING: Your password has expired.\nPassword change required but no TTY available."}

  • Alternatively, the following log events appear within the SDDC Manager at /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log:

    YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[#####]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:###
    YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[#####]:         timeout = setTimeout(reject, sshCommandTimeout, new Error(failureMessage));
    YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[#####]: Error: Remote ssh command timed out
    YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[#####]:     at /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:###:##
    YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[#####]:     at new Promise (<anonymous>)
    YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[#####]:     at process.processTicksAndRejections (node:internal/process/task_queues:##:#)
    YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[#####]: Node.js v18.20.4
    YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[#####]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:###
    YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[#####]: Error: Remote ssh command timed out

  • When this issue occurs, VCF Fleet management might fail to fetch tasks and present the following error: Retrieving task counts failed. Http failure response for https://<Ops_FQDN>/vcf-operations/plug/sddc-###/plugin/api/v1/tasks?pageSize=1: 503 OK



  • In the VCF Operations UI, the SDDC Manager tile (under Administration) only loads a blank white screen.

Environment

  • VMware Cloud Foundation (VCF) 5.x
  • VMware Cloud Foundation (VCF) 9.x

Cause

This issue occurs if the SDDC Manager fails to establish an SSH connection with the Management Domain vCenter during bootup. Common causes include:

  • Deactivated SSH on the Management Domain vCenter Server.
  • Expired root password on the Management Domain vCenter.
  • The root password is changed natively on the vCenter without updating the credentials in the SDDC Manager database.

Resolution

  1. Enable SSH on the Management Domain vCenter Server Enable or Disable SSH and Bash Shell Access
  2. Restart the sddc-manager-ui-app service: systemctl restart sddc-manager-ui-app
  3. If Step 2 does not resolve the issue, change the root password on the Management Domain vCenter and update the SDDC Manager with the new credentials using the steps below.

    1. Log in to the SDDC Manager as root and generate an API token:
      TOKEN=$(curl -d '{"username" : "<sso username>", "password" : "<sso password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')

    2. Use the generated token to query the system for the vCenter root credentials and locate its ID:
      curl -X GET 'localhost/v1/system/credentials?entityType=VCENTER&credentialType=SSH' -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN"

    3. Push the new password into the SDDC Database (replace `<id from above step>` and `<new password>` with the actual values):
      curl -X PUT 'localhost/v1/system/credentials/<id from above step>' -d '<new password>' -H "Content-type:application/json" -H "Authorization: Bearer $TOKEN"| json_pp

    4. Restart the sddc-manager-ui-app service:
      systemctl restart sddc-manager-ui-app

    5. If the SDDC Manager UI remains inaccessible after completing the above steps, execute a full restart of the vCenter Server management services, clear the web browser's cache, and attempt to re-establish the connection in a new or incognito window.

Additional Information

Powered by Wolken Software