Troubleshooting Storage Issues in vRealize Operations

Products

• VMware Aria Suite

Issue/Introduction

Various storage related issues may arise in vRealize Operations.
To assist in troubleshooting, see the below list of the most common storage related articles.

Symptoms:
You may experience one or more the the below symptoms.

• The vRealize Operations UI login page displays:

Data retriever is not initialized yet.

• The vRealize Operations cluster is down.
• The vRealize Operations Admin UI reports:

The Cluster was shut down because one node was out of disk space.

• vRealize Operations shows the following alert(s):
  • Disk space on node is low   
  • Fsdb is running critically low on disk space
  • Fsdb is estimated to run out of disk space soon
• Running the df -h command on the vRealize Operations nodes shows one or more files systems above 80% used.

Resolution

To verify which volume is out of space, run the df -h command on each vRealize Operations node.  See the Related Information section for more information.

/storage/db is out of space

• Increase the size of the /storage/db partition: Adding additional storage to a node in vRealize Operations (2016022)
  • Adding disk larger than 2TB: Disks larger than 2TB do not increase the /storage/db volume in vRealize Operations (2135465)
• Clean up space in /storage/db partition: Safely cleaning /storage/db space in vRealize Operations (2093903)
• If the cluster was deployed through the vSphere Content Library: vRealize Operations 8.x deployed from content library is out of disk space (82624)

  /storage/log is out of space

• Clean up logs from failed log rotation: /storage/log is full on vRealize Operations (53289)
• Clean up space in /storage/log partition: Safely cleaning /storage/log space in vRealize Operations (2145578)

 / is out of space

• Clean up space in the / partition: / is full on vRealize Operations Manager 8.0 (76154)

Additional Information

To run the df -h command on a vRealize Operations node, follow the steps below.

1. Log into the Primary node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
2. Run the following command:

df -h

3. You will see output including the below volumes:

Filesystem             Size  Used Avail Use% Mounted on
...
/dev/sda4               16G  4.5G   11G  30% /
...
/dev/mapper/data-db    207G   25G  172G  13% /storage/db
/dev/mapper/data-core   20G  2.3G   17G  92% /storage/core
/dev/mapper/data-log    20G  7.5G   12G  41% /storage/log
...

Impact/Risks:
To avoid cluster stability issues, do NOT expand an existing disk.