VCHA state degraded with alert: Appliance configuration is out of sync. Automatic failover is not allowed. Manual failover is allowed.

Products

• VMware vCenter Server 8.0

Issue/Introduction

•  VCHA getting alert: Appliance configuration is out of sync. Automatic failover is not allowed. Manual failover is allowed.  
• :

/var/log/vmware/vcha/vcha.log - reports below error

• YYYY-MM-DDTHH:MM:SSZ error vcha [02538] [Originator@6876 sub=ReplFacade-smal1Frp] Sync failed: /etc/vmware-rhttpproxy/endpoints. conf.d/vdtc.conf, event: 2097156
YYYY-MM-DDTHH:MM:SSZ verbose vcha[02538] [Originator@6876 sub=VchaUtil] Executing system command; /usr/bin/rsync, args: [ -- recursive, -- checksum, -- perms, -- times, -- group, -- owner, -- links, -- protect-args, -- te
mp-dir=/storage/vcha/.tmpfiles, -- info=progress, -- timeout=60, -- rsh=ssh -i /home/vcha/. ssh/id_rsa -o UserKnownHostsFile=/home/vcha/. ssh/known_hosts, /etc/vmware-rhttpproxy/endpoints. conf.d/vmonapi-proxy. conf, vc
ha@IP_ADDRESS:/etc/vmware-rhttpproxy/endpoints.conf.d/]
YYYY-MM-DDTHH:MM:SSZ info vcha[03686] [Originator@6876 sub=vpxUtil] System command failed; '/usr/bin/rsync', args: [ -- recursive, -- checksum, -- perms, -- times, -- group, -- owner, -- links, -- protect-args, -- temp-di
r=/storage/vcha/.tmpfiles, -- info=progress, -- timeout=60, -- rsh=ssh -i /home/vcha/.ssh/id_rsa -o UserKnownHostsFile=/home/vcha/.ssh/known_hosts, /etc/vmware-rhttpproxy/endpoints.conf.d/vlcm-proxy. conf, vcha@IP_ADDRESS:/etc/vmware-rhttpproxy/endpoints.conf.d/], exit code: 2
-- > stdout:
-- > stderr:
> VMware vCenter Server 8.0.3.00000
-- >
-- > Type: vCenter Server with an embedded Platform Services Controller
-- >
> sudo: Account or password is expired, reset your password and try again
-- > sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
> protocol version mismatch -- is your shell clean?
-- > (see the rsync manpage for an explanation)
protocol incompatibility (code 2) at compat.c(608) [sender=3.2.4]
YYYY-MM-DDTHH:MM:SSZ error vcha [03686] [Originator@6876 sub=RsyncRepl-smallFrp] Rsync failed, retcode: 2,
-- > VMware vCenter Server 8.0.3.00000> Type: vCenter Server with an embedded Platform Services Controller                                                                                                                                                                                                                                       ---> sudo: Account or password is expired, reset your password and try again
> sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
-- > protocol version mismatch -- is your shell clean?
-- > (see the rsync manpage for an explanation)
protocol incompatibility (code 2) at compat.c(608) [sender=3.2.4]rsync error

Environment

vCenter 8.x

Cause

This issue occurs due to root password sync issue between active and passive node. Root password was changed post deploying VCHA in active node which has not been replicated to passive node.

Resolution

1. Try to change root password for passive vCenter using below command (use the same root password as in active node)
    sudo passwd root
    sudo faillock --reset --user root (if account is locked)
2. Disable and Enable VCHA for Appliance sync.

3. If still issue persists, disable and remove VCHA
3. Shutdown and Delete from disk passive and witness nodes
4. Re-deploy VCHA 

Additional Information

Deploying vCenter High Availability