Aria Operations cluster failed to go offline

Products

• VMware Aria Suite

Issue/Introduction

While bringing vROps cluster offline from Admin UI, you may get the error

Cluster failed to go offline

 

Other Symptoms

• One or more nodes stuck in status Waiting for Analytics when taking cluster offline from Admin UI.

Environment

• vRealize Operations 8.x
• Aria Operations 8.x

Cause

There are many potential causes for this, and they are usually environmental:

• Old snapshots - You should not keep Aria Operations running on snapshots for more than 72 hours, as this results in performance issues.
• Disks full

Resolution

Ensure that you have attempted Force Take Cluster Offline from Admin UI 2 to 3 times before proceeding with the next steps below.

 

As cluster cannot go offline manually we need to stop the services manually.

1. Log in to Aria Operations console on affected node as root user using SSH, or console in vCenter.
   
   
2. Stop the main service using command
   

   service vmware-vcops stop

    

3. The Admin UI will show the node as offline after some time.

Additional Information

You may also want to verify whether the node slice is set to offline, you can do this with command:

vrops-status

 

The first line will output the node slice online/offline status, similar to Slice Online-true -or- Slice Online-false.

• If the command returns Online-false, the node status is correctly set to Offline.
• If the command returns Online-true, the node status has not been set to Offline correctly. 

If the result is Online-true, ensure that the status is set to Online-false by issuing the following command:

sed -i -e 's/sliceonline = true/sliceonline = false/' /usr/lib/vmware-vcopssuite/utilities/sliceConfiguration/data/roleState.properties

 

This can be done to ensure that the cluster services are not automatically started if node is rebooted.