Aria Automation or Orchestrator patching fails with "WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!" after replacing a missing node in the cluster
Article ID: 410138
Updated On:
Products
Issue/Introduction
When attempting to patch a clustered VMware Aria Automation or Orchestrator environment after a missing or deleted node has been replaced, the patching process may fail.
The upgrade logs contain errors indicating a failure to verify SSH connectivity between the nodes.
Specifically, the /var/log/vmware/prelude/upgrade-error-<date>.log on the primary node shows messages similar to the following:
[ERROR][2025-09-11 15:46:34][<node-primary>.<domain>][Exit Code: 255] Synced failed of upgrade data with node: <node-new>.<domain>.
[ERROR][2025-09-11 15:46:36][<node-primary>.<domain>][Exit Code: 1] Attempt failed to run command: /opt/scripts/upgrade/ssh-verify-peers.sh.
Pseudo-terminal will not be allocated because stdin is not a terminal.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
...
Host key verification failed.
...
[ERROR][2025-09-11 15:46:36][<node-primary>.<domain>] Remote command failed: /opt/scripts/upgrade/ssh-verify-peers.sh at host: <node-new>.<domain>
[ERROR][2025-09-11 15:46:36][<node-primary>.<domain>][Exit Code: 1] Verification that nodes are able to connect to one another and to this node failed.
Environment
Aria Automation / Orchestrator 8.18.x
Cause
This issue occurs because the original nodes in the cluster retain the SSH key fingerprint of the old, deleted node in their /home/root/.ssh/known_hosts file.
When a new node is deployed with the same hostname or IP address as the old one, it has a new, different SSH key.
During the patch's SSH verification step, the existing nodes detect a mismatch between the cached fingerprint and the new fingerprint presented by the replacement node.
This is interpreted as a potential man-in-the-middle attack, causing the SSH connection to fail and halting the upgrade.
Resolution
The stale SSH key fingerprint for the replaced node must be removed from the /home/root/.ssh/known_hosts file on all other existing nodes in the cluster, and the new key must be accepted.
Follow these steps on each of the original, existing nodes in the cluster.
1. SSH to an existing node in the cluster (e.g., the primary node).
2. Attempt to SSH to the new/replaced node to confirm the host key verification error appears.
root@<node-primary> [ ~ ]# ssh <node-new>.<domain>
You should see the WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! message.
3. Remove all references to the new node's old fingerprint from the known_hosts file using the ssh-keygen command. Replace <node-new>.<domain> with the FQDN of the replaced node.
root@<node-primary> [ ~ ]# ssh-keygen -R <node-new>.<domain>
# Host <node-new>.<domain> found: line 3
# Host <node-new>.<domain> found: line 5
# Host <node-new>.<domain> found: line 6
# Host <node-new>.<domain> found: line 7
# Host <node-new>.<domain> found: line 8
/home/root/.ssh/known_hosts updated.
Original contents retained as /home/root/.ssh/known_hosts.old
4. SSH to the new node again. You will be prompted to accept the new key fingerprint. Type yes and press Enter. You can use Ctrl+c to exit after the password prompt appears.
root@<node-primary> [ ~ ]# ssh <node-new>.<domain>
The authenticity of host '<node-new>.<domain> (<Affected Node IP>)' can't be established.
ED25519 key fingerprint is SHA256:<fingerprint>.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '<node-new>.<domain>' (ED25519) to the list of known hosts.
Welcome to VMware Aria Automation Orchestrator Appliance 8.18.1
root@<node-new>.<domain>'s password:
5. Verify that you are no longer prompted to accept the key by SSHing to the new node one last time. You can use Ctrl+c to exit at the password prompt.
root@<node-primary> [ ~ ]# ssh <node-new>.<domain>
Welcome to VMware Aria Automation Orchestrator Appliance 8.18.1
root@<node-new>.<domain>'s password:
6. As a cleanup step, remove the backup known_hosts.old file created by the ssh-keygen command.
root@<node-primary> [ ~ ]# rm /home/root/.ssh/known_hosts.old
7. Repeat steps 1-6 on all other remaining original nodes in the cluster.
Once the new node's SSH key has been accepted on all other nodes, you may retry the upgrade procedure.
Feedback
