Failover Backups fail with error : Host key verification failed

Products

• VMware Smart Assurance

Issue/Introduction

Failover backup actions fail with error "Host key verification failed"

This happens only during the SCP operation even if the password less communication between the Failover Manager and Remote host works fine. 

Environment

All Supported Smarts versions

Cause

The SCP function used during the backup actions fails with the error specified which is seen on debug logs with "-v" option in place.

Example Log:

Executing: program /usr/bin/ssh host <Remote Host>, user smarts, command scp -v -r -p -f /opt/InCharge/IP/smarts/local/repos/icf/<Domain>.rps
OpenSSH_8.0p1, OpenSSL 1.1.1k  FIPS 25 Mar 2021^M
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Connecting to <Remote Host> [Remote Host IP Address],  port 22.
debug1: Connection established.
debug1: identity file /home/<user>/.ssh/id_rsa type 0
debug1: identity file /home/<user>/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug1: Authenticating to <Remote Host>:22 as '<user>'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: [email protected]
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: [email protected]
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: kex: curve25519-sha256 need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:<Host Key>
Host key verification failed.

Resolution

• Backup the existing copy of known_hosts file under: /home/<user>/.ssh/
• Run the command

ssh-keyscan -t ecdsa-sha2-nistp256 -H <Remote Host>

• Copy the key with cryptographic algorithm : ecdsa-sha2-nistp256 
• Paste the copied key under: /home/<user>/.ssh/known_hosts for the <Remote Host> entry 
• Run the backup actions again to confirm the changes take effect successfully. 
• If the issues persist, please raise a case with Broadcom Support for further review. 

Additional Information

Example Entries :

cat /home/<user>/.ssh/known_hosts
<Remote Host>,<Remote Host IP> ecdsa-sha2-nistp256 <HOST KEY IS WRITTEN HERE ON KNOWN_HOSTS FILE>

ssh-keyscan -t ecdsa-sha2-nistp256 -H <Remote Host IP>
# <Remote Host IP>:22 SSH-2.0-OpenSSH_8.0 |1|<Encrypted String> ecdsa-sha2-nistp256 <HOST KEY IS PRESENTED HERE>