VMware Aria Operations High Availability (HA) stuck in "Failed to Deactivate" state

Products

• VMware Aria Suite

Issue/Introduction

• The purpose of the KB is to help troubleshoot the HA to fix cluster stability.

Symptoms:

• The High Availability in Aria Operations is stuck in the "Failed to Deactivate" state which restricts any configuration change in the cluster like Certificate Renewal.
• The Activate/Deactivate HA button is greyed out.
• The cluster can still be in an Online state but it will not perform precisely.

Environment

VMware Aria Operations 8.x

Cause

• It can happen if the services fail to either Activate or Deactivate the HA.

Resolution

1. Take snapshots of all nodes. For steps refer https://knowledge.broadcom.com/external/article?legacyId=2149269 
2. SSH to all the nodes using the root credentials.
3. Stop casa on all the nodes by running the command: service vmware-casa stop
4. Open casa.db.script in text edit by using the command: vi /data/db/casa/webapp/hsqldb/casa.db.script
5. Identify the ha_transition_state in the last line of the file
6. Change the value from "FAILED_TO_DEACTIVATE' to "NONE" on all the nodes. (Please note the above screenshot has been taken from Aria Operations version 8.12. The value will be "FAILED_TO_DISABLE" in earlier versions.)
7. Save and close the file.
8. Start casa on all the nodes by running the command: service vmware-casa start
9. Bring the cluster online from the admin UI.

Additional Information

How to enable/disable HA when the option is not available in Aria Operations Manager UI

Impact/Risks:

• It will impact the stability of the cluster which will in turn restrict configuration changes like Certificate renewal.