Error: Cluster failed to go offline when bringing cluster online in VMware Aria Operations

Products

• VCF Operations/Automation (formerly VMware Aria Suite)
• VMware Telco Cloud Operations

Issue/Introduction

Symptoms:

1. The cluster fails to come online.
   
   
2. All nodes appear to be in an online and running state at the VM level.
   
   
3. Events are triggered in external ticketing systems regarding cluster failure state.
   
   
4. The VMware aria operations UI displays the error: Cluster failed to go offline.
   
   

Environment

VMware Aria Operations 8.18.5

VMware Telco Cloud Platform (TCP) 5.0

Cause

A host-level storage disruption or insufficient free space on the datastores prevents the cluster from completing the metadata synchronization required to transition to an online state.

On the vCenter, navigate to the cluster and select "Datastores." The affected datastores will display alerts or warnings related to insufficient storage space.

Resolution

Steps:

1. Forcefully take the Aria Operations cluster offline via the Admin UI.

2. Perform a graceful power-off sequence on all cluster nodes through vCenter.

3. Verify datastore health and ensure no alerts for "insufficient storage space" exist in vCenter.

4. Perform a power-on sequence on all cluster nodes.

5. Bring the cluster back online.

For detailed sequence instructions, see Shutdown and Startup sequence for VCF/Aria Operations cluster

Additional Information

Aria Operations Cluster Failed to Come Online Due to Incorrect is_admin Node Configuration on casa.db.script