Activating VCF Operations HA stuck in a loop after adding replica and data nodes

Products

• VCF Operations

Issue/Introduction

Administrators may encounter an issue where activating VMware Cloud Foundation (VCF) Operations High Availability (HA) is stuck in a continuous loop.

When a replica and data node are added to a single-node VCF Operations 9.1 deployment via the Build UI, the nodes show as running and online but have different versions. The cluster status is Online, but the cluster initialization remains in progress for an extended period, and HA continuously shows as activating.

Environment

VCF Operations 9.1.x

Cause

The UI does not provide an option to select the product version during node addition and automatically pulls the latest build from the depot. As a result, HA cannot activate because the VCF Operations nodes are running on different versions.

Resolution

This is a known issue and VCF Engineering team is working on a fix in a future release. It is recommended to Subscribe to a Broadcom knowledge article by article or product to be updated on fix status. 

Workaround: 

1. Take snapshots of all nodes including Cloud proxies in vCenter .

2. Bring the slice(node) offline and update the casa service to offline on all nodes as mentioned in the resolution section of High Availability stuck in activating state with new deployed replica node KB article.

3. Manually edit the configuration script to set ha_transition_state to "NONE" on all nodes as as mentioned in the resolution section of VMware Aria Operations High Availability (HA) stuck in "Failed to Deactivate" state or stuck in "Activating" state  KB article.

4. Disable HA using below command :

   curl -i -k -u 'admin:<password>' -X POST https://<primary_node_IP>/casa/config/cluster/ha -d '{"is_enabled":false}' -H "Accept: application/json" -H "Content-type: application/json"

   Where:

5. • password is the password of the local admin account.
   • primary_node_IP is the IP addresses of the primary node

6. Remove the two newly added nodes (the ones on the newer version) from the Admin UI.

7. Delete the corresponding virtual machines from VMware vCenter Server.

8. Upgrade Fleet services to latest version so that you can add new nodes later.

9. Upgrade VCF Operations to version latest version.

10. Scale out the VCF Operations cluster again by adding a Replica and Data node from the Build Lifecycle page.

 

Additional Information

Note: The task 'Add Replica node to component VCF Operations' may fail due to a timeout on the SDDC Manager side. However, in the Admin UI, you will see that HA is activating and cluster initialization is in progress. Wait a few hours for HA to activate and the cluster to come online. Afterward, you can retry the failed task, and it will complete successfully.