High Availability stuck in activating state with new deployed replica node
search cancel

High Availability stuck in activating state with new deployed replica node

book

Article ID: 402019

calendar_today

Updated On:

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

  • Enabling HA with a newly deployed replica node is stuck in the Activating state, the cluster is in a going online state, and the node status is Waiting for Analytics.
  • When signing into a root SSH session for the newly deployed replica node, the prompt displays root@localhost.
  • Running the hostname command in a root SSH session returns localhost.
  • Viewing the /etc/hosts file on the node shows localhost.
  • The /storage/log/vcops/log/casa.log file reports a name resolution error.
  • Running nslookup for the FQDN of the node fails, as does the reverse nslookup.

Environment

Aria Operations 8.18.x

Cause

Incorrect vAPP value configured on VM such as FQDN or DNS

Resolution

Option 1: Snapshots created before enabling HA

  1. Revert to snapshots.
  2. Set the value of the Domain Name property and the Domain Name Servers property to the correct values using the document Configure vApp Properties as a guide. Please power on the nodes as per KB Shutdown and Startup sequence for Aria Operations cluster.
  3. Bring the cluster back online from the Admin UI.
  4. Enable HA.

Option 2: Snapshots not created before enabling HA

  1. Log in with an SSH session using the root account for all nodes in the cluster (Primary/Replica/Data).
  2. Run the following command on all cluster nodes (Primary/Replica/Data) to stop cluster services on the node and bring to an offline state:
    $VMWARE_PYTHON_BIN $VCOPS_BASE/../vmware-vcopssuite/utilities/sliceConfiguration/bin/vcopsConfigureRoles.py --action bringSliceOffline --offline "Disable HA"
  3. Take a snapshot of all nodes in the cluster as per KB Snapshot Creation in VMware Aria Operations.
  4. Disable HA using the KB How to enable/disable HA when the option is not available in Aria Operations UI.
  5. If the cluster status is still showing that HA is activating, please apply KB VMware Aria Operations High Availability (HA) stuck in "Failed to Deactivate" state or stuck in "Activating" state.
  6. Set the value of the Domain Name property and the Domain Name Servers property to correct values using the document Configure vApp Properties as a guide. Please power on the nodes as per KB Shutdown and Startup sequence for Aria Operations cluster.
  7. Update the casa.db.script to Offline status using the following command for each node:
    service vmware-casa stop;sleep 10;cp --backup=t /storage/db/casa/webapp/hsqldb/casa.db.script /storage/db/casa/webapp/hsqldb/casa.db.script.backup;sed -ri 's/"onlineState":"\w+"/"onlineState":"OFFLINE"/g;s/"initialization_state":"\w+"/"initialization_state":"NONE"/g;s/"online_state":"\w+"/"online_state":"OFFLINE"/g;s/"online_state_reason":"\w+"/"online_state_reason":""/g;s/"remove_node_state":"\w+"/"remove_node_state":"NONE"/g;s/"installation_state":"\w+"/"installation_state":"DONE"/g' /storage/db/casa/webapp/hsqldb/casa.db.script;sleep 2;service vmware-casa start;echo -e "\e[1;35mCluster Status has been modified\e[0m";grep "onlineState" /storage/db/casa/webapp/hsqldb/casa.db.script;
  8. Bring the cluster back online through the UI.
  9. Enable HA.

Additional Information

If the issue persists after completing the steps above please generate a support bundle as per doc Create a VMware Aria Operations Support Bundle .  Revert to snapshots and raise a technical SR as per kb Creating and managing Broadcom support cases then upload the logs to the portal as per kb Uploading files to cases on the Broadcom Support Portal

Powered by Wolken Software