VCF Workload Domain deployment fails at "Validate vMotion Network Connectivity"

Products

• VMware SDDC Manager / VCF Installer
• VMware Cloud Foundation

Issue/Introduction

• Workload domain deployment in VMware Cloud Foundation (VCF) fails during the "Validate vMotion Network Connectivity" phase. The SDDC Manager UI reports a failure to connect to ESXi hosts using the specified credentials, despite valid credentials being provided.
• Error on SDDC UI : 

  Validate vMotion Network ConnectivityProgress Messages:Validate vMotion Network ConnectivityError:Message: Validate vMotion Network ConnectivityRemediation Message:Reference Token: #####Cause: Failed to connect to ESX Host <host fqdn> with the specified credentials, Failed to connect to ESX Host <host1 fqdn> with the specified credentials, Failed to connect to ESX <host2 fqdn> with the specified credentials, Failed to connect toESX Host <host3 fqdn> with the specified credentials - Product version VCF 9.0

• Error in /var/log/vmware/vcf/domainmanager/domainmanager.log

  ERROR [vcf_dm,##########,adec] [c.v.evo.sddc.common.util.SshUtil,####]  Unable to create jsch CLI session:com.jcraft.jsch.JSchException: HostKey has been changed: <hostname>
  ERROR [vcf_dm,##########,adec] [c.v.e.s.c.u.c.SshCommandExecuter,####]  Could not connect to the SSH server @<hostname> for configuration.com.jcraft.jsch.JSchException: HostKey has been changed: <hostname>
  ERROR [vcf_dm,##########,adec] [c.v.evo.sddc.common.util.SshUtil,####]  Unable to create jsch CLI session:com.jcraft.jsch.JSchException: java.net.ConnectException: Connection refused    at com.jcraft.jsch.Util.createSocket(Util.java:394)        at com.jcraft.jsch.Session.connect(Session.java:215) 

Environment

VMware Cloud Foundation (VCF) 9.0

Cause

The failure is caused by an SSH Host Key mismatch between the SDDC Manager and the ESXi hosts. This happens when the SSH fingerprint stored in the SDDC Manager’s known_hosts does not match the current fingerprint presented by the ESXi host, causing the security handshake to fail and blocking automated configuration tasks.

Resolution

Regenerate the host keys for the host listed in the logs to fix the SDDC connectivity issue with the host

1. Create a virtual machine snapshot of the SDDC Manager before proceeding.
2. Remediate the incorrect/mismatched Host Keys stored in SDDC manager by following the procedure mentioned in Broadcom KB 316028.
3. Once the keys are synchronized and the trust relationship is re-established, return to the SDDC Manager UI and Retry the Workload Domain deployment task.