Running bosh ssh from jumpbox server fails with "Host key verification failed"

Products

• Operations Manager

Issue/Introduction

After switching from Ubuntu jumpbox to Red Hat Enterprise Linux (RHEL) 7, the bosh ssh (BOSH SSH) command stops working. cf SSH and direct SSH to any server, including the BOSH VM and Ops Manager VM, works as expected. However,  bosh ssh does not work through the BOSH CLI and you get the following error message:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING #####!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is SHA256:Ikr<REDACTED>hyk.
Please contact your system administrator.
Add correct host key in /home/xxxxx/.bosh/tmp/ssh-known-hosts264265227 to get rid of this message.
Offending RSA key in /home/xxxxx/.bosh/tmp/ssh-known-hosts264265227:1 ECDSA host key for xx.xx.xx.xx has changed and you have requested strict checking. Host key verification failed. Running SSH: 1 error occurred: *

Running command: 'ssh -tt -o ServerAliveInterval=30 -o ForwardAgent=no -o PasswordAuthentication=no -o IdentitiesOnly=yes -o IdentityFile=/home/xxxxx/.bosh/tmp/ssh-priv-key072503388 -o UserKnownHostsFile=/home/xxxxx/.bosh/tmp/ssh-known-hosts264265227 -o StrictHostKeyChecking=yes xx.xx.xx.xx -l bosh_3129624d3a924a9', stdout: '', stderr: '':

exit status 255 " 

Environment

Product Version: 2.10

Resolution

In order to get more detailed output, increase the verbosity of BOSH CLI commands:

export BOSH_LOG_LEVEL=debug

Then run the following command:

bosh -d <deployment> ssh <instance> --opts="-o StrictHostKeyChecking=no"

Where --opts="-o StrictHostKeyChecking=no" overrides the default setting. Confirm whether the connection is successful.

To disable debug verbosity, run the following command:

export BOSH_LOG_LEVEL=none

Lastly, follow the steps listed below:

1. Verify the /home/xxxxx/.bosh/tmp/ssh-known-hostsxxxx that was once connected.

2. Compare this with Ops Manager when connecting to the same server with bosh ssh. Both should have the same host key.

3. If the host key is the same, it could be that the host key algorithm configured in your jump box is different from rsa-sha2-512, which is the only one that BOSH CLI supports. This can be checked by running ssh -vvv <host IP> and then by looking for the following line:

...
debug1 (3317986): kex: host key algorithm: ecdsa-####-nistp256
...

The jumpbox default can be overwritten by specifying algorithms in ~/.ssh/config:

Host *
HostKeyAlgorithms rsa-####-512,rsa-####-256