Error: VMware Cloud Foundation is initializing - SDDC Manager UI hangs during upgrade (VMware Cloud Foundation 9.x)

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• The SDDC Manager becomes inaccessible during the upgrade from VCF 9.0.1 to 9.1

• The following error messages appear in /var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log:

YYYY-MM-DD INFO [common,##############,####] [c.v.e.s.c.s.a.r.CommonServicesSamlMetadataExchangeController,http-nio-127.0.0.1-####-exec-##] Start getPscRootSshCredential(), Psc host: [VC-FQDN]
YYYY-MM-DD INFO [common,##############,####] [c.v.e.s.i.s.PscInventoryServiceImpl,http-nio-127.0.0.1-####-exec-##] Get Psc - ########-####-####-####-########
YYYY-MM-DD INFO [common,#################,####] [c.v.e.s.a.u.api.rest.SshController,http-nio-127.0.0.1-####-exec-##] getSshKnownHostsConfiguration(): host = [VC-FQDN]
YYYY-MM-DDINFO [common,###############,####] [c.vmware.vcf.secure.ssh.SshExecuter,http-nio-127.0.0.1-####-exec-##] Mina sshd: Establishing SSH session to host: [VC-FQDN]
YYYY-MM-DD ERROR [common,#################,####] [c.v.v.s.s.c.CustomKnownHostsServerKeyVerifier,sshd-SshClient[########]-nio2-thread-1] Server at [VC-FQDN]:22 presented wrong key type: ecdsa-sha2-nistp256 and key: ########################################################################
YYYY-MM-DD ERROR [common,##############,####] [c.v.v.s.ssh.common.SshClientImpl,http-nio-127.0.0.1-####-exec-##] Failed to open ssh connection to [VC-FQDN] in 30 seconds.
YYYY-MM-DD0 ERROR [common,############,####] [c.vmware.vcf.secure.ssh.SshExecuter,http-nio-127.0.0.1-####-exec-##] error message: [VC-FQDN]: Server key did not validate
YYYY-MM-DD ERROR [common,############,####] [c.vmware.vcf.secure.ssh.SshExecuter,http-nio-127.0.0.1-####-exec-##] Host key validation failed for host [VC-FQDN]
com.vmware.vcf.secure.ssh.common.SshClientException: [VC-FQDN]: Server key did not validate

• The following error messages appear in /var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log:

YYYY-MM-DD -ERROR SDDC-MANAGER-UI-APP ##### - [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:56] axios.error.response.status {"status":500}
YYYY-MM-DD -ERROR SDDC-MANAGER-UI-APP ##### - [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:57] axios.error.response.headers {"headers":{"cache-control":"no-cache, no-store, max-age=0, must-revalidate","connection":"cl
ose","content-type":"application/json","date":"Day, Date Month YYYY HH:MM:SS GMT","expires":"0","pragma":"no-cache","server":"nginx","x-content-type-options":"nosniff","x-frame-options":"DENY","x-xss-protection":"0"}}
YYYY-MM-DD -ERROR SDDC-MANAGER-UI-APP ##### - [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:58] axios.error.response.data {"data":"Failed to establish SSH session to [VC-FQDN] as the host key valid
ation failed"}

Environment

VMware Cloud Foundation (VCF) 9.x

Cause

• This issue occurs because of a case-sensitivity mismatch between the vCenter FQDN entries stored in the SDDC Manager known_hosts files and the FQDN/PNID on the vCenter Server.
• In VCF 9.x, if the vCenter Server uses an uppercase FQDN but the SDDC Manager stores the host keys in lowercase, the commonsvcs API fails to find a matching trusted key, leading to an SSH validation failure.

Resolution

1. Run the fix_known_hosts.sh script  SSH connection fails with "reject HostKey" or "Primary psc init failed" errors
2. If the known_hosts files on the SDDC Manager do not contain the correctly cased FQDN after following Step 1, manually update the files on the SDDC Manager appliance to match the exact casing of the vCenter FQDN.

1. 1. Log in to the SDDC Manager appliance as root via SSH.
   2. Identify the correct casing of the vCenter FQDN (for example, by checking the vCenter PNID or DNS entry).
   3. Manually edit the following four files to ensure the vCenter entries use the correct casing:
      1. /root/.ssh/known_hosts
      2. /etc/vmware/vcf/commonsvcs/known_hosts
      3. /home/vcf/.ssh/known_hosts
      4. /opt/vmware/vcf/commonsvcs/defaults/hosts/known_hosts
   4. Save the changes to all four files.
   5. Restart the SDDC Manager appliance by running the following command: reboot
   6. After the appliance restarts, log back in to the SDDC Manager UI and retry the upgrade.