Getting "Failed to setup VCHA network config. Logs available at /var/log/vmware/vcha" when trying to setup vCenter HA

Products

• VMware vCenter Server

Issue/Introduction

• When trying to setup vCenter HA from vSphere UI, the task immediately fails with error "Failed to setup VCHA network config. Logs available at /var/log/vmware/vcha".
  
  
• In /var/log/vmware/vpxd/vpxd.log of the affected appliance, the below errors confirm the same symptom.
  
  error vpxd[] [Originator@6876 sub=Default opID=<ID>] [VpxLRO] -- ERROR task-<ID>--  -- Datacenters -- com.vmware.vcenter.vcha.cluster.deploy: :vmodl.fault.SystemError
--> Result:
--> (vmodl.fault.SystemError) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    reason = "Failed to setup VCHA network config. Logs available at /var/log/vmware/vcha"
-->    msg = ""
--> }
--> Args:
-->
  
  
• In /var/log/vmware/vcha/prepare-vcha.log, the HA preparation task can be seen failing because the "vchaeth.cfg.tpl" file is missing.
  
  INFO prepare-vcha IPV6 enabled on eth1. Enabling multicast all mode
INFO prepare-vcha Enabled multicast all mode for eth1
INFO prepare-vcha *** SUCCESS: Vcha pre-setup ***
INFO prepare-vcha *** START: Vcha pre-setup ***
INFO prepare-vcha embedded DB Type can be vcha enabled.
INFO prepare-vcha Found eth1 interface with IP <IP-Address>
ERROR prepare-vcha Failed to setup Vcha Network config
Traceback (most recent call last):
  File "/usr/lib/vmware-vcha/scripts/preSetup.py", line 699, in setupVchaEthCfg
    createVchaEthCfg(selfIp, selfFailoverIp, peerIp, peerFailoverIp,
  File "/usr/lib/vmware-vcha/scripts/createVchaEthCfg.py", line 193, in createVchaEthCfg
    raise IOError('vchaeth.cfg.tpl not found.')
OSError: vchaeth.cfg.tpl not found.

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x
VMware vCenter Server 9.x

Cause

One or more system files inside /etc/vmware-vcha of the vCenter Server Appliance are missing.

• Below is how it looks like in a setup where VCHA is enabled.
  
  root@vcsa [ /etc/vmware-vcha ]# ls -l
total 44
-rw-r--r-- 1 vpxd root 2181 Nov 18 20:54 alarmSpec.json
-rw-r--r-- 1 root root   82 Nov 18 20:58 alt1.nw
-rw-r--r-- 1 root root   36 Nov 18 20:58 nodeuuid
drwxr-xr-x 5 root root 4096 Nov 18 20:58 ssl
-rwxr-xr-x 1 root root    4 Nov 18 20:58 vcha.aware
-rw-r--r-- 1 vpxd root 1406 Nov 18 20:54 vcha.cfg.tpl
-rw-r--r-- 1 root root  739 Nov 18 20:58 vchaeth.cfg
-rw-r--r-- 1 vpxd root  706 Nov 18 20:55 vchaeth.cfg.tpl
-rw-r--r-- 1 vpxd root  859 Nov 18 20:55 vchaspec.properties
-rw-r--r-- 1 root root  548 Jul 15 10:18 vmware-vmon.service.bak
-rw-r--r-- 1 root root  556 Nov 18 20:58 vmware-vmon.service.vcha
root@vcsa [ /etc/vmware-vcha ]#


• Below is how it looks like in a setup where VCHA is not enabled.
  
  root@vcsa [ /etc/vmware-vcha ]# ls -l
total 16
-rw-r--r-- 1 vpxd root 2181 Nov 18 20:54 alarmSpec.json
-rw-r--r-- 1 vpxd root 1406 Nov 18 20:54 vcha.cfg.tpl
-rw-r--r-- 1 vpxd root  706 Nov 18 20:55 vchaeth.cfg.tpl
-rw-r--r-- 1 vpxd root  859 Nov 18 20:55 vchaspec.properties
root@vcsa [ /etc/vmware-vcha ]#


• In an affected or problematic system, below is how it might look like.
  
  root@vcsa [ / ]# cd /etc/vmware-vcha/
root@vcsa [ /etc/vmware-vcha ]# ls -l
total 0
root@vcsa [ /etc/vmware-vcha ]#

Resolution

1. SSH to another vCenter Server Appliance of the same version and build.
   
   
2. Copy all the files from its /etc/vmware-vcha directory to the affected appliance.
   
   
3. Login to the affected vCenter Server Appliance.
   
   
4. Correct the permissions on the copied files from root:root to vpxd:root. The command to do the same shall be below.
   
   chown vpxd:root <file_name>


5. Post this, re-attempt the VCHA deployment.