VCHA configuration reports failure with error: vCenter HA appliance out of sync

Products

VMware vCenter Server

Issue/Introduction

Configuring vCenter High Availability (VCHA) via UI fails with "Appliance state is out of sync."
/var/log/vmware/vcha/vcha.log you will see similar entries:

_{YYYY-MM-DDTHH:MM:SS.691+02:00 verbose vcha[206416] [Originator@6876 sub=VchaUtil] Executing system command; /usr/bin/rsync, args: [--recursive,--checksum,--perms,--times,--group,--owner,--links,--protect-args,--temp-dir=/storage/vcha/.tmpfiles,--info=progress,--timeout=60,--rsh=ssh -i /home/vcha/.ssh/id_rsa -o UserKnownHostsFile=/home/vcha/.ssh/known_hosts,/storage/db/vmware-vmdir/snapshot/data.mdb,vcha@<passive node IP>:/storage/db/vmware-vmdir/snapshot/]}

_{YYYY-MM-DDTHH:MM:SS.747+02:00 info vcha[206416] [Originator@6876 sub=vpxUtil] System command failed; '/usr/bin/rsync', args: [--recursive,--checksum,--perms,--times,--group,--owner,--links,--protect-args,--temp-dir=/storage/vcha/.tmpfiles,--info=progress,--timeout=60,--rsh=ssh -i /home/vcha/.ssh/id_rsa -o UserKnownHostsFile=/home/vcha/.ssh/known_hosts,/storage/db/vmware-vmdir/snapshot/data.mdb,vcha@<passive node IP>:/storage/db/vmware-vmdir/snapshot/], exit code: 255}
_{--> stdout:}
_{--> stderr:}
_{--> VMware vCenter Server 8.0.2.00300}
_-->
_{--> Type: vCenter Server with an embedded Platform Services Controller}
_-->
_{--> Connection closed by "<passive node IP>" port 22}
_{--> rsync: connection unexpectedly closed (0 bytes received so far) [sender]}
_{--> rsync error: unexplained error (code 255) at io.c(228) [sender=3.2.4]}
_-->
_{YYYY-MM-DDT10:34:20.747+02:00 error vcha[206416] [Originator@6876 sub=RsyncRepl-largeFrp] Rsync failed, retcode: 255, error:}
_{--> VMware vCenter Server 8.0.2.00300}
_-->
_{--> Type: vCenter Server with an embedded Platform Services Controller}
_-->
_{--> Connection closed by "<passive node IP>" port 22}
_{--> rsync: connection unexpectedly closed (0 bytes received so far) [sender]}
_{--> rsync error: unexplained error (code 255) at io.c(228) [sender=3.2.4]}

_{YYYY-MM-DDTHH:MM:SS.750+02:00 verbose vcha[206705] [Originator@6876 sub=Cluster opID=WorkQueue-4f7cfba8] Received ack=true from "<passive node IP>" for kvstore (version 4294967332)}
_{YYYY-MM-DDTHH:MM:SS.750+02:00 error vcha[206416] [Originator@6876 sub=SnmpAgent] Failed to create trp tmp file /var/spool/snmp/1718181260_6_350_6876}
_{--> N7Vmacore23FileIONotFoundExceptionE(Could not find file : /var/spool/snmp/1718181260_6_350_6876)}
/var/log/vmware/vcha/sshConnect.log

YYYY-MM-DDTHH:MM:SS.225Z WARNING sshConnect retry attempt 34 failed [Errno None] Unable to connect to port 22 on <passive node IP>
YYYY-MM-DDTHH:MM:SS.230Z INFO sshConnect Retry attempt 34
YYYY-MM-DDTHH:MM:SS.288Z WARNING sshConnect retry attempt 35 failed [Errno None] Unable to connect to port 22 on <passive node IP>
YYYY-MM-DDTHH:MM:SS.293Z INFO sshConnect Retry attempt 35

Environment

vCenter Server 8.0

Cause

SSH auth fails.

Resolution

To fix this issue,

Create private/public SSH-keypair for root-user

To generate key with default algorithm run following cmd.

ssh-keygen

OR,

You may choose from following algorithms

ssh-keygen -t dsa | ecdsa | ed25519 | rsa
The public key must be placed in authorized_keys of root-user

cat /root/.ssh/id_ed25519.pub >> /root/.ssh/authorized_keys
Edit SSH config and update preferred auth method.

/etc/ssh/sshd_config

This needs to be added

PreferredAuthentications publickey,password,keyboard-interactive,gssapi-with-mic
Disable VCHA
Reboot VC
Re-enable VCHA

Additional Information

Note: Requires reboot of vCenter and reconfiguration of VCHA.

# cd ~/.ssh/

# cat authorized_keys
# ssh-keygen

Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub

# cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

# vi /etc/ssh/sshd_config (Paste the keys to sshd_config for example ssh-keygen -t dsa | ecdsa | ed25519 | rsa) and save the file.

Disable to vcha and rebooted the vCenter.