SDDC Manager UI stuck in "VMware Cloud Foundation is initializing" state due to SSH host key validation failure

Products

• VCF Operations

Issue/Introduction

The SDDC Manager UI is completely inaccessible, persistently displaying the message: "VMware Cloud Foundation is initializing..."

Management operations and workload deployments are blocked (Severity 2 Operational Degradation).

Reviewing the SDDC Manager UI application logs (/var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log) reveals the following recurring exception:

axios.error.response.data {"data":"Failed to establish SSH session to as the host key validation failed"}

Environment

VCF Operations 9.0x

Cause

This issue occurs when the SDDC Manager's secure validation registries fall out of synchronization with the Management vCenter's SSH fingerprint.

In this specific scenario, the mismatch was triggered by a chain of events:

1. Storage Deadlock: The NFS mount (/nfs/vmware/vcf/nfs-mount) reached 100% capacity due to stale NSX backup files (cluster-node-backups), placing the SDDC Manager in a deadlocked state.

2. Out-of-Band Password Rotation: During this deadlock, the Management vCenter root password expired and was updated directly on the vCenter appliance.

3. Fingerprint Mismatch: Updating the password rotated the host SSH key/fingerprint. Because the SDDC Manager was deadlocked, it was unable to capture and synchronize the new fingerprint across its backend Postgres databases and internal known_hosts files, breaking the commonsvcs API communication.

Resolution

Note: Do not attempt to resolve this by manually appending keys via OS-level commands (e.g., ssh-keyscan and ssh-keygen -R). Manual modification of /root/.ssh/known_hosts, /home/vcf/.ssh/known_hosts, and /etc/vmware/vcf/commonsvcs/known_hosts alters required file permissions, bypasses application-level API alignments, and misses a critical internal fourth known_hosts file required by the VCF architecture.

 

To resolve the issue:

1. Clear the Deadlock: Ensure the underlying storage issue is resolved. Prune stale files (e.g., backups older than 30 days) from /nfs/vmware/vcf/nfs-mount to restore adequate capacity.

2. Align Credentials: If the vCenter credentials still require alignment, use the SDDC REST API (PUT localhost/v1/system/credentials/<ID>) to register the newly rotated password.

3. Execute the Official Remediation Script: Download and execute the official Broadcom SSH remediation Python script (fixHostKeys.py) or shell script (fix_known_hosts.sh for non-FIPS environments) on the SDDC Manager appliance. See kb: 316028

4. Restart Services: The script will properly iterate through all required backend databases and file registries to align the fingerprints. Once complete, restart the SDDC Manager services to restore UI access.

Additional Information

Related Information / References:

KB 407083: Identifies and troubleshoots the specific sddcManagerServer.log host key validation failure error preventing UI initialization.

KB 442142: Details the internal VCF architecture requirement for SSH keys, including the undocumented fourth known_hosts file utilized by the commonsvcs API, which explains why manual OS-level updates fail.

KB 316028: Provides the official Broadcom recommendation and the necessary scripts (fixHostKeys.py / fix_known_hosts.sh) required to safely resolve SSH host key mismatches without breaking backend VCF dependencies.