Access to SDDC Manager UI is lost because of Management vCenter Server Key validation failed

Products

VMware SDDC Manager / VCF Installer

Issue/Introduction

Logs indicate a validation failure of the vCenter host keys:

/var/log/vmware/vcf/commonsvcs/vcf-commonsvcs.log

ERROR [common,0000000000000000,0000] [c.v.v.s.s.c.CustomKnownHostsServerKeyVerifier,sshd-SshClient[3959167b]-nio2-thread-1] Server at <vc_fqdn>/<vc_ip_address>:22 presented wrong key type: 
ecdsa-sha2-nistp256 and key: AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBdz6DK1HTjs/GtM6qpPMqP661iHnIxvCAEz/2yUEjedUWyxIGB2kEMeq6MuHmh9v4qUbeorR+Sdq9EIiVUW2I8=
ERROR [common,<task_id>] [c.v.v.s.ssh.common.SshClientImpl,http-nio-127.0.0.1-7100-exec-7] Failed to open ssh connection to <vc_fqdn> in 30 seconds.
ERROR [common,<task_id>] [c.vmware.vcf.secure.ssh.SshExecuter,http-nio-127.0.0.1-7100-exec-7] error message: <vc_fqdn>: Server key did not validate
ERROR [common,<task_id>] [c.vmware.vcf.secure.ssh.SshExecuter,http-nio-127.0.0.1-7100-exec-7] Host key validation failed for host <vc_fqdn>
mon.SshClientException: <vc_fqdn>: Server key did not validate
..
Caused by: org.apache.sshd.common.SshException: Server key did not validate
..
ERROR [common,<task_id>] [c.v.e.s.c.s.a.r.CommonServicesSamlMetadataExchangeController,http-nio-127.0.0.1-7100-exec-7] Exception in shellAgnosticCommandExec for host <vc_fqdn> : 
com.vmware.vcf.secure.ssh.errors.VcfSshException: Failed to establish SSH session to <vc_fqdn> as the host key validation failed
ERROR [common,<task_id>] [c.v.e.s.c.s.a.r.CommonServicesSamlMetadataExchangeController,http-nio-127.0.0.1-7100-exec-7] Get error in getIDPMetadataFromVC
com.vmware.vcf.secure.ssh.errors.VcfSshException: Failed to establish SSH session to <vc_fqdn> as the host key validation failed

/var/log/vmware/vcf/sddc-manager-ui-app/sddcManagerServer.log

ERROR SDDC-MANAGER-UI-APP 1501368 - [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:58] axios.error.response.data {"data":"Failed to establish SSH session to <vc_fqdn> as the host key validation failed"}
ERROR SDDC-MANAGER-UI-APP 1501368 - [services/wrappers/requestPromiseWrapper.js, init-pscs-primary, logAxiosError:68] axios.error.stack {"stack":"AxiosError: Request failed with status code 500

SDDC manager UI stuck on Loading page:

Environment

VMware Cloud Foundation 9.x
VMware Cloud Foundation 5.x

Cause

SDDC Manager requires a connection to the Management vCenter in order to retrieve the inventory and SSO configuration required for the authentication process.
Failing to establish an SSH connection will result in the services not been able to query those information.
A host key mismatch on the know hosts file of the SDDC Manager is one of the causes that can result in a failure to establish an SSH connection to the vCenter.

Resolution

To validate if the vCenter host key is correct or we have a mismatch, follow the steps below:

Get the current SSH key from vCenter

ssh-keyscan -H <vCenter-FQDN> >> /tmp/current_vc_key

Compare with SDDC Manager’s known_hosts

cat /home/vcf/.ssh/known_hosts | grep -i <vCenter-FQDN>

or

grep <vCenter-FQDN> /home/vcf/.ssh/known_hosts

Compare the fingerprints using output of below 2 commands:

check the file generated in step 1:
```
ssh-keygen -lf /tmp/current_vc_key
```
Check the exisiting entries:
```
ssh-keygen -lf <(grep <vCenter-FQDN> /home/vcf/.ssh/known_hosts)
```
Once compared, you should find mismatch in key.

Below outpout shows matching hostky in sddcmanager:

root@<SDDC-Manager> [ /home/vcf ]# ssh-keygen -lf /tmp/current_vc_key

256 SHA256:LDXXXXXXXXXXXXXXXXXXXXXXXXXXXXX (ED25519)
3072 SHA256:b2a18QXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX (RSA)
256 SHA256:+QKUszXXXXXXXXXXXXXXXXXXXXXXXXXXXXX (ECDSA)

root@<SDDC-Manager> [ /home/vcf ]# ssh-keygen -lf < ( grep <vCenter-FQDN> /home/vcf/.ssh/known hosts)

256 SHA256:+QKUsz5XXXXXXXXXXXXXXXXXXXXXXXXX <vCenter-FQDN> (ECDSA)
3072 SHA256:b2a18QXXXXXXXXXXXXXXXXXXXXXX <vCenter-FQDN> (RSA)

If the host keys values are mismatched, use the resolution steps of the KB article below to create new entries:

How to update the SSH host keys on the SDDC Manager

Additional Information

Troubleshooting SDDC Manager UI access Issues