SDDC Manager UI Fails to Load with error "VMware Cloud Foundation is initializing"

Products

• VMware SDDC Manager
• VMware vCenter Server

Issue/Introduction

• Breaking of Enhanced Linked Mode (ELM) between two management VC nodes of two different VCF instances, SDDC Manager UI fails to load.

• From /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log observed below error

YYYY-MM-DDT22:30:27.818+0000 WARN [027725c9910042c2, 832cc6500c2e4024] [services/pscUtils.js, init-pscs, attemptPSCInitWithRetry:111]
100.109: VError: PSC Initilization attempt "11" failed: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500
    at Object.initializationPscError (/opt/vmware/vcf/sddc-manager-ui-app/server/src/errors/VCFError.js:104:5)
    at attemptPSCInitWithRetry (/opt/vmware/vcf/sddc-manager-ui-app/server/src/services/pscUtils.js:104:46)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Error Info: {"retryCount":11,"status":403,"errorModule":100,"errorCode":109,"code":"ERR_BAD_RESPONSE"}
caused by:
100.108: VError: Failed to initiate PSC: Primary psc init failed and failover psc init also failed: Unable to retrieve iDP Metadata: Request failed with status code 500

• From /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log | grep error observed SSH connection error

YYYY-MM-DDT01:58:52.715+0000 ERROR [common,fb4ccbf573f045f2,680e] [c.v.e.s.c.s.a.r.CommonServicesSamlMetadataExchangeController,http-nio-127.0.0.1-7100-exec-3] Exception in shellAgnosticCommandExec for host <VCSA_FQDN> : com.vmware.vcf.secure.ssh.errors.VcfSshException: Failed to establish SSH session to <VCSA_FQDN>

com.vmware.vcf.secure.ssh.errors.VcfSshException: Failed to establish SSH session to <VC_FQDN>

YYYY-MM-DDT01:58:53.863+0000 ERROR [common,fb4ccbf573f045f2,680e] [c.v.e.s.c.s.a.r.CommonServicesSamlMetadataExchangeController,http-nio-127.0.0.1-7100-exec-3] Exception in shellAgnosticCommandExec for host <VCSA_FQDN> : com.vmware.vcf.secure.ssh.errors.VcfSshException: Failed to establish SSH session to <VCSA_FQDN>

Environment

SDDC Manager 5.x

VMware vCenter Server

Cause

SSH connection from SDDC manager to vCenter fails due to SSH Host Keys stored in SDDC manager was Blank. SDDC Manager is unable to SSH into the node(s) in question to run through the attempted workflow due to a mismatch in what it expects the Host key to be vs what is being presented by the node.

Connect to Platform DB,

• • Connect to the SDDC Platform database:
    •         psql -h localhost -U postgres -d platform

• • # select * from vcenter ;
                  id                  | creation_time | modification_time | bundle_repo_datastore | datastore_name  | ssh_host_key | ssh_host_key_type | status |
   type    |       version        |          vm_hostname           | vm_management_ip_address |  vm_name   | join_sso_status
--------------------------------------+---------------+-------------------+-----------------------+-----------------+----------

#########-5550-43da-ae36-###########| ######## | lcm-bundle-repo       | vSAN_DS_Name |              |                   | ACTIVE|
MANAGEMENT | 8.0.3.00300-#######| vCSA_FQDN> | ##.##.###.##| vCSA_FQDN>|

Resolution

Followed below steps to resolve the issue,

• Update the SSH Host Key , by running fixHostKeys.py. KB 316028.

• If issue still persists, change and update new Password by following steps from KB 389716