SDDC Manager password remediation fails with "Failed to execute command, error : SSH: Failed to establish SSH session"
search cancel

SDDC Manager password remediation fails with "Failed to execute command, error : SSH: Failed to establish SSH session"

book

Article ID: 396540

calendar_today

Updated On:

Products

VMware Cloud Foundation

Issue/Introduction

The purpose of this KB is to fix the SDDC manager password remediation when fails with "SSH: Failed to establish SSH session" by updating the host keys on the SDDC manager.

  • Failed to establish SSH connection to the service connected to the SDDC.
  • Following errors are reported : 

/var/log/vmware/vcf/operationsmanager.log

YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,019c84c0869d7578,76b9] [c.v.e.s.c.u.c.SshCommandExecuter,om-exec-28] Could not connect to the SSH server @ <FQDN of the service> for configuration.
com.jcraft.jsch.JSchException: reject HostKey: <FQDN of the service>
        at com.jcraft.jsch.Session.checkHost(Session.java:789)
        at com.jcraft.jsch.Session.connect(Session.java:345)

YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,019c84c0869d7578,76b9] [c.v.v.p.h.LinuxHostHelperService,om-exec-28] Exception when testing host credential scom.vmware.evo.sddc.common.util.command.CommandExecuterException: SSH: Failed to establish SSH session to <FQDN of the service>
Caused by: com.jcraft.jsch.JSchException: reject HostKey: <FQDN of the service>


YYYY-MM-DDTHH:MM:SS ERROR [vcf_om,019c84c0869d7578,76b9] [c.v.v.p.u.c.AbstractPasswordChanger,om-exec-28] Failed to execute command, error : SSH: Failed to establish SSH session to <fqdn of the service>
com.vmware.vcf.passwordmanager.exception.PasswordUpdateException: Failed to execute command, error : SSH: Failed to establish SSH session to <FQDN of the service>.
        at com.vmware.vcf.passwordmanager.update.changers.SshPasswordChanger.doSSHTest(SshPasswordChanger.java:79)
        at com.vmware.vcf.passwordmanager.update.changers.SshPasswordChanger.doTest(SshPasswordChanger.java:35)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.updateAsync(AbstractPasswordChanger.java:429)
        at com.vmware.vcf.passwordmanager.update.changers.AbstractPasswordChanger.doUpdate(AbstractPasswordChanger.java:198)
        at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:100)
        at com.vmware.vcf.passwordmanager.rotate.AbstractPasswordTransactionExecutor$1.call(AbstractPasswordTransactionExecutor.java:88)
        at org.springframework.cloud.sleuth.instrument.async.TraceCallable.call(TraceCallable.java:67)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: com.vmware.evo.sddc.common.util.command.CommandExecuterException: SSH: Failed to establish SSH session to <FQDN of the service>
        at com.vmware.evo.sddc.common.util.command.SshCommandExecuter.<init>(SshCommandExecuter.java:55)

  • SSH connection to the service from SDDC manager will also fail :

root@<SDDC> [ /var/log/vmware/vcf/operationsmanager ]# ssh root@<fqdn of the service>
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:VPQydz/BQatHFX+5M4ly5UeQiQGQ/sh1JRsDyEwftKw.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending RSA key in /root/.ssh/known_hosts:49
Host key for <fqdn of the service> has changed and you have requested strict checking.
Host key verification failed.

 

Environment

VMware Cloud Foundation 4.x

VMware Cloud Foundation 5.x

Cause

Host Keys can be changed on a node for a variety of reasons, including but not limited to:

  • Restore from a backup
  • Manual rebuild
  • Manual intervention to change the Host Key

As a result of this change key, SDDC Manager is unable to SSH into the node(s) in question to run through the attempted workflow due to a mismatch in what it expects the Host key to be vs what is being presented by the node.

Resolution

Update the host keys using the fixHostkeys.py or fix_known_hosts.sh by following the below article :

How to update the SSH host keys on the SDDC Manager

Additional Information

Impact/Risks:
MINIMAL: The script edits 4 known_hosts files, namely:

/root/.ssh/known_hosts
/etc/vmware/vcf/commonsvcs/known_hosts
/home/vcf/.ssh/known_hosts
/opt/vmware/vcf/commonsvcs/defaults/hosts/known_hosts

Since this is a not a major change, the risk is minimal. That being said, since entries are being removed and added, a snapshot of the SDDC Manager is highly recommended, in the event we need to revert to a state from before the script was run.

Powered by Wolken Software