Heapdumps filling /storage/db in Aria Operations

Products

• VMware Aria Suite

Issue/Introduction

This article provides the safe procedure to remove old heapdumps on /storage/db in Aria Operations.

Note: If this article does not resolve your issue, see Troubleshooting Storage Issues in vRealize Operations (83239).

Symptoms:

• The Aria Operations user interface is inaccessible.
• In the Admin UI, the status of all nodes is Waiting for Analytics to Start.
• The /storage/db file system on one or more nodes is full.
• Receiving an error "Cluster was shut down because  ####.example.com node were out of disk space".

Environment

Aria Operations 8.x

Cause

/storage/db file system on one or more nodes becomes full when old heapdumps are still retained.

Resolution

Impact/Risks:
It is imperative to take a snapshot following How to take a Snapshot of vRealize Operations (2149269) before proceeding with the steps below.

 

If there have been services crashes on a vRealize Operations Node, undesirable Java heap files can consume space.  Make sure there is not current heapdump and remove only oldest heap files. This will help regaining some space and start investigating the service crash using the latest heapdump if any.

1. Log into the Aria Operations node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
2. Run the following command to find the Java heap files:

find /storage/db -name \*.hprof -exec ls -lah {} \;

Note: When running the preceding command, ensure that there is a space between {} and \; as you may get this error:
find: missing argument to '-exec'.

3. Files that are found using the above statement can be removed with the rm command:

rm filename

Note: Replace filename with the full path to the files found.

Example: rm /storage/db/vcops/heapdump/java_pid9754.hprof

4. Repeat steps 1-3 on any other Aria Operations nodes as needed.

Note: If no files are identified that can be removed, additional virtual disk space should be added to all nodes in the cluster.  See Adding additional storage to a node in vRealize Operations (2016022) for more information.