Troubleshooting Storage Issues in Aria Operations

Products

• VMware Aria Suite

Issue/Introduction

Various storage related issues may arise in Aria Operations.
To assist in troubleshooting, see the below list of the most common storage related articles.

Symptoms:
You may experience one or more the the below symptoms.

• The Aria Operations UI login page displays:

Data retriever is not initialized yet.

• The Aria Operations cluster is down.
• The Aria Operations Admin UI reports:

The Cluster was shut down because one node was out of disk space.

• Aria Operations shows the following alert(s):
  • Disk space on node is low   
  • Fsdb is running critically low on disk space
  • Fsdb is estimated to run out of disk space soon
  • vRealize Operations Cluster database is running out of disk space

 

• Running df -h command on the Aria Operations nodes shows one or more files systems above 80% used.

Resolution

To verify which volume is out of space, run the df -h command on each Aria Operations node.  See the Related Information section for more information.

/storage/db is out of space

• Increase the size of the /storage/db partition: Adding additional storage to a node in Aria Operations (Formerly vRealize Operations)(318768)
  • Adding disk larger than 2TB: Disks larger than 2TB do not increase the /storage/db volume in vROps(342739)
• Clean up space in /storage/db partition: Heapdumps filling /storage/db in Aria Operations (317811)
• If the cluster was deployed through the vSphere Content Library: VMware Aria Operations 8.x deployed from content library is not functioning (324369)

/storage/log is out of space

• Clean up logs from failed log rotation: /storage/log is full on vRealize Operations (318019)
• Clean up space in /storage/log partition: How to safely clean /storage/log in vRealize Operations (315972)
• Clean up excessive logging for local6: /storage/log partition full when Local6 logs are not rotated (314221)

/ is out of space

• Clean up space in the / partition: / is full on vRealize Operations Manager 8.0 (310960)

Additional Information

To run the df -h command on a Aria Operations node, follow the steps below.

1. Log into the Primary node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
2. Run the following command:

df -h

3. You will see output including the below volumes:

Filesystem             Size  Used Avail Use% Mounted on
...
/dev/sda4               16G  4.5G   11G  30% /
...
/dev/mapper/data-db    207G   25G  172G  13% /storage/db
/dev/mapper/data-core   20G  2.3G   17G  92% /storage/core
/dev/mapper/data-log    20G  7.5G   12G  41% /storage/log
...

Impact/Risks:
To avoid cluster stability issues, do NOT expand an existing disk.