SDDC Manager upgrade from 9.0.1 to 9.1 hangs with UI stuck in 'Initializing' state

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

SDDC Manager is inaccessible during upgrade from VCF 9.0.1 to 9.1

/var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log 

YYYY-MM-DD INFO [common,d40bd54768bd473f,8025] [c.v.e.s.c.s.a.r.CommonServicesSamlMetadataExchangeController,http-nio-127.0.0.1-7100-exec-11] Start getPscRootSshCredential(), Psc host: [VC-FQDN]
YYYY-MM-DD INFO [common,d40bd54768bd473f,8025] [c.v.e.s.i.s.PscInventoryServiceImpl,http-nio-127.0.0.1-7100-exec-11] Get Psc - b0a01398-3ab9-4c94-828a-12cb0c1a8a23
YYYY-MM-DD INFO [common,6a1190fb50eb4533b0d61c59fabd48b8,2f5d] [c.v.e.s.a.u.api.rest.SshController,http-nio-127.0.0.1-7100-exec-3] getSshKnownHostsConfiguration(): host = [VC-FQDN]
YYYY-MM-DDINFO [common,d40bd54768bd473f,8025] [c.vmware.vcf.secure.ssh.SshExecuter,http-nio-127.0.0.1-7100-exec-11] Mina sshd: Establishing SSH session to host: [VC-FQDN]
YYYY-MM-DD ERROR [common,0000000000000000,0000] [c.v.v.s.s.c.CustomKnownHostsServerKeyVerifier,sshd-SshClient[390c21bb]-nio2-thread-1] Server at [VC-FQDN]:22 presented wrong key type: ecdsa-sha2-nistp256 and ke
y: ########################################################################
YYYY-MM-DD ERROR [common,d40bd54768bd473f,8025] [c.v.v.s.ssh.common.SshClientImpl,http-nio-127.0.0.1-7100-exec-11] Failed to open ssh connection to [VC-FQDN] in 30 seconds.
YYYY-MM-DD0 ERROR [common,d40bd54768bd473f,8025] [c.vmware.vcf.secure.ssh.SshExecuter,http-nio-127.0.0.1-7100-exec-11] error message: [VC-FQDN]: Server key did not validate
YYYY-MM-DD ERROR [common,d40bd54768bd473f,8025] [c.vmware.vcf.secure.ssh.SshExecuter,http-nio-127.0.0.1-7100-exec-11] Host key validation failed for host [VC-FQDN]
com.vmware.vcf.secure.ssh.common.SshClientException: [VC-FQDN]: Server key did not validate
        at com.vmware.vcf.secure.ssh.common.SshClientImpl.connect(SshClientImpl.java:175)

/var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log

YYYY-MM-DD -ERROR SDDC-MANAGER-UI-APP 31382 - [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:56] axios.error.response.status {"status":500}
YYYY-MM-DD -ERROR SDDC-MANAGER-UI-APP 31382 - [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:57] axios.error.response.headers {"headers":{"cache-control":"no-cache, no-store, max-age=0, must-revalidate","connection":"cl
ose","content-type":"application/json","date":"Sat, 23 May 2026 11:38:17 GMT","expires":"0","pragma":"no-cache","server":"nginx","x-content-type-options":"nosniff","x-frame-options":"DENY","x-xss-protection":"0"}}
YYYY-MM-DD -ERROR SDDC-MANAGER-UI-APP 31382 - [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:58] axios.error.response.data {"data":"Failed to establish SSH session to [VC-FQDN] as the host key valid
ation failed"}

Environment

VCF 9.x 

Cause

This issue occurs because of a case-sensitivity mismatch between the vCenter FQDN entries stored in the SDDC Manager known_hosts files and the FQDN/PNID on the vCenter Server.

In VCF 9.x, if the vCenter Server uses an uppercase FQDN but the SDDC Manager has recorded the host keys in lowercase, the commonsvcs API fails to find a matching trusted key, leading to an SSH validation failure.

Resolution

1. Run the fix_known_hosts.sh script  SSH connection fails with "reject HostKey" or "Primary psc init failed" errors

• Incase if the 4 known_hosts files on the SDDC manager are not updated with correct case sensitive FQDN

2. To resolve this, manually update the known_hosts files on the SDDC Manager appliance to match the correct casing of the vCenter FQDN.

• Log in to the SDDC Manager appliance as root via SSH.
• Identify the correct casing of the vCenter FQDN (e.g., check the vCenter PNID or DNS entry).
• Manually edit the following four files to ensure the vCenter entries use the correct casing:
  • /root/.ssh/known_hosts
  • /etc/vmware/vcf/commonsvcs/known_hosts
  • /home/vcf/.ssh/known_hosts
  • /opt/vmware/vcf/commonsvcs/defaults/hosts/known_hosts
• Save the changes to all four files.
• Restart the SDDC Manager services or reboot the appliance: reboot
• After the appliance restarts, log back into the SDDC Manager UI and retry the upgrade.