Failed to init PSC and/or Postgres. Check the SDDC Manager UI Server logs for more details."VMware Cloud Foundation is initializing"/var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log:YYYY-MM-DDT HH:MM:SS VERBOSE [a4c53bab982c432c, 3878912b2d1849e4, d7705f9eba5a4aa5] [services/logging/opentrace.js, init-pscs-primary, _finish:231] Span init-pscs-primary complete: Total time: 2511ms {"duration":2511}100.109: VError: PSC Initialization attempt "6" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500 at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)Error Info: {"retryCount":6,"status":403,"errorModule":100,"errorCode":109,"code":"ERR_BAD_RESPONSE"}caused by:100.108: VError: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500 at Object.initiatePscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)Error Info: {"errorModule":100,"errorCode":108,"code":"ERR_BAD_RESPONSE"}YYYY-MM-DDT HH:MM:SS ERROR [c2f2b47783674dbc, f81e250ca5e64fc2, 3e9563ebd619463c] [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:58] axios.error.response.data {"data":"Error executing remote command via SSH: WARNING: Your password has expired.\nPassword change required but no TTY available."}/var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log:YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: timeout = setTimeout(reject, sshCommandTimeout, new Error(failureMessage));YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: Error: Remote ssh command timed outYYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: at /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275:65YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: at new Promise (<anonymous>)YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5)YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: Node.js v18.20.4YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: /opt/vmware/vcf/sddc-manager-ui-app/server/src/services/sso-initialization.js:275 ^YYYY-MM-DDT HH:MM:SS <SDDC Manager Hostname> sddc-manager-ui-app[70645]: Error: Remote ssh command timed outRetrieving task counts failed. Http failure response for https://<Ops_FQDN>/vcf-operations/plug/sddc-###/plugin/api/v1/tasks?pageSize=1: 503 OKThis issue occurs if the SDDC Manager fails to establish an SSH connection with the Management Domain vCenter during bootup. Common causes include:
systemctl restart sddc-manager-ui-app# TOKEN=$(curl -d '{"username" : "<sso username>", "password" : "<sso password>"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')# curl -X GET 'localhost/v1/system/credentials?entityType=VCENTER&credentialType=SSH' -H 'Content-Type: application/json' -H "Authorization: Bearer $TOKEN"# curl -X PUT 'localhost/v1/system/credentials/<id from above step>' -d '<new password>' -H "Content-type:application/json" -H "Authorization: Bearer $TOKEN"| json_pp# systemctl restart sddc-manager-ui-appThe error "Failed to initiate PSC: Primary psc init failed and failover psc init also failed" will also be logged if the vCenter server SSH Host Keys stored in SDDC manager contain a mismatch.
To resolve this mismatch, run the script from the article How to update the SSH host keys on the SDDC Manager to update the host keys of vCenter in SDDC.
See KB, In SDDC Manager, vCenter Upgrade Precheck fails with the error: "vCenter root account is locked or has expired" if ED25519 has been enabled on the vCenter server.