SSH connection to VKS supervisor fails with "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!"

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

When attempting to ssh on to the vks supervisor you see the below error:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
<old fingerprint>
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending RSA key in /root/.ssh/known_hosts:2
Host key for <supervisor ip>has changed and you have requested strict checking.
Host key verification failed.

Environment

vSphere Kubernetes Service

Cause

This issue occurs because the original nodes in the cluster retain the SSH key fingerprint of the old, deleted node in their /home/root/.ssh/known_hosts file.

When a new node is deployed with the same hostname or IP address as the old one, it has a new, different SSH key.

During the patch's SSH verification step, the existing nodes detect a mismatch between the cached fingerprint and the new fingerprint presented by the replacement node.

This is interpreted as a potential man-in-the-middle attack, causing the SSH connection to fail and halting the login.

Resolution

The stale SSH key fingerprint for the replaced node must be removed from the /home/root/.ssh/known_hosts file on all other existing nodes in the cluster, and the new key must be accepted.
Follow these steps on each of the original, existing nodes in the cluster.

1. SSH to the vCenter

2. Attempt to SSH to the vks supervisor to confirm the host key verification error appears.

 [ ~ ]# ssh root@<ip address>
You should see the WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! message.

3. Remove all references to the new node's old fingerprint from the known_hosts file using the ssh-keygen command. Replace <node-new>.<domain> with the FQDN of the replaced node.

[ ~ ]# ssh-keygen -R <ip address>

# Host <ip address> found: line 1
# Host <ip address> found: line 2
/home/root/.ssh/known_hosts updated.
Original contents retained as /home/root/.ssh/known_hosts.old

4. SSH to the new node again. You will be prompted to accept the new key fingerprint. Type yes and press Enter. You can use Ctrl+c to exit after the password prompt appears.

 [ ~ ]# ssh root@<ip address>

The authenticity of host '<ip address>' can't be established.
ECDSA key fingerprint is <fingerprint>.
This host key is known by the following other names/addresses:
    ~/.ssh/known_hosts:5: <ip address>
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '<ip address>' (ECDSA) to the list of known hosts.
Connection reset by <ip address> port 22

5. Verify that you are no longer prompted to accept the key by SSHing to the new node one last time. You can use Ctrl+c to exit at the password prompt.
 [ ~ ]# ssh root@<ip address>