Upgrading ESXi hosts from VCF Operations using Fleet Management fails at precheck with error "ESXi CLI checks for legacy boot mode and host filesystem have failed"

Products

• VMware vSphere ESXi

Issue/Introduction

• Upgrading ESXi hosts via Fleet Management in VCF Operations fails at the precheck stage with the error:
  
  Description: Perform ESXi checks through ESXi CLI for legacy boot mode and host filesystem symlinks
  Error Message: ESXi CLI checks for legacy boot mode and host filesystem symlinks have failed
  Remediation: Please inspect the operationsmanager log file for further details on the ESXi CLI command failures. Check whether the host <HOST_FQDN> is using legacy boot mode using: 'vsish -e cat /hardware/firmwareType'. Check whether there are broken symlinks on the host using: 'find -L / -maxdepth 1 -user root -type l'
  
  
• On the SSH session of the ESXi host, running the suggested commands result in the following output:
  
  
  • #vsish -e cat /hardware/firmwareType
    Output: 1 -> UEFI
  • #find -L / -maxdepth 1 -user root -type l
Output:
    
    
• On the SDDC Manager, in /var/log/vmware/vcf/operationsmanager/operationsmanager.log:

YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,################,####] [c.v.e.s.c.u.c.SshCommandExecuter,precheck-validation1] Could not connect to the SSH server @ <HOST_FQDN> for configuration.
com.jcraft.jsch.JSchException: HostKey has been changed: <HOST_FQDN>
        at com.jcraft.jsch.Session.checkHost(Session.java:###)
        at com.jcraft.jsch.Session.connect(Session.java:###)
        at com.vmware.evo.sddc.common.util.SshUtil.getSession(SshUtil.java:###)
        at com.vmware.evo.sddc.common.util.SshUtil.getSession(SshUtil.java:###)
        at com.vmware.evo.sddc.common.util.command.SshCommandExecuter.<init>(SshCommandExecuter.java:61)


• SSH to the impacted ESXi host from the SDDC Manager fails with the below error:
  
  # ssh root@<HOST_FQDN>
  
  @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
#########
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /root/.ssh/known_hosts:30
Host key for <HOST_FQDN> has changed and you have requested strict checking.
Host key verification failed.

Environment

VMware Cloud Foundation 9.x

Cause

The issue occurs due to the SSH host key (fingerprint) mismatch between the SDDC Manager and the impacted ESXi host, which prevents the SDDC Manager from establishing an SSH session to the ESXi host to run the prechecks.

Resolution

Update the SSH host key on the SDDC Manager by following the KB article: How to update the SSH host keys on the SDDC Manager