vCenter High Availability failover from the passive node back to the active node leaves VCHA inoperable.

Products

• VMware vCenter Server

Issue/Introduction

• vCenter HA was able to failover the passive node, but failing back over to the primary leaves the passive offline.
• A snapshot was taken on one or more VCHA nodes while VCHA was enabled which led to failed failover
• During the failed failover, vMon restarts the passive node without the "HACore" profile in /var/log/vmware/vmon/vmon-syslog.log

    notice vmon  Starting vMon with profile 'NONE'

• There may errors referring to the prefreeze script being unable to create a partnerlist file in /var/log/preefreeze/prefreeze_#####_#_stderr.log on the VCHA nodes

ERROR:root:Failed to create partnerlist file at the location /storage/db/vmware-vmdir/tmp/.
Stdout Error: Backup requires vmdird service running

Stderr
Traceback (most recent call last):
  File "/usr/lib/vmware/site-packages/create-partner-list.py", line 69, in <module>
    main()
  File "/usr/lib/vmware/site-packages/create-partner-list.py", line 62, in main
    port, domainName = GetDCConfig()
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/util/Common.py", line 826, in GetDCConfig
    lsObj = LookupServiceClient(vmafdClient.GetLSLocation(), retry_count=2)
RuntimeError: unidentifiable C++ exception
 

Cause

When a snapshot is taken of vCenter, part of the prefreeze script creates a file called backupMarker.txt, but the prefreeze script fails due to VCHA being enabled. During the second failover, the passive node is rebooted. During the boot, the applmgmt service finds the /etc/vmware/backupMarker.txt file and attempts to perform restore. As part of the restore, it masks all services, which prevents the vcha service from coming up.

Resolution

Taking snapshots of a vCenter Server with VCHA enabled is not supported and the vCenter should be restored from backup. Furthermore before attempting a failover operation, ensure that no snapshots exist and remove the /etc/vmware/backupMarker.txt file on any of the vCenter Server HA nodes.

Workaround:

To restore the vCenter, follow the below steps:

1. Repair the Network Configuration:

   1. Log in to the vCenter Console (ALT+F1)  for the VCHA node that seems to contain both the IP addresses (Management IP as well as VCHA IP) by accessing the ESXi host clients
   2. Navigate to the network directory: cd /etc/systemd/network/
   3. Check for the manual file: ls 10-eth0.network.manual
   4. Rename the file back to the standard configuration: mv 10-eth0.network.manual 10-eth0.network
   5. Restart the network service: systemctl restart systemd-networkd

2. Clean up VCHA and Files:

   1. Remove the stale VCHA configuration: vcha-destroy -f
   2. Remove the backup marker file if present: rm /etc/vmware/backupMarker.txt
   3. Remove the HA Nic from the VCHA active node and remove the passive and witness nodes after powering them down

3. Unmask and Start Services:

   1. Unmask all services: systemctl list-unit-files | grep masked | awk '{print $1}' | xargs -n1 systemctl unmask
   2. Restart all vCenter services: service-control --stop --all && service-control --start --all

Additional Information

FAQ: vCenter High Availability (2148003)