Troubleshooting Storage Issues in Aria Operations

Products

• VMware Aria Suite

Issue/Introduction

Various storage related issues may arise in Aria Operations.
To assist in troubleshooting, see the below list of the most common storage related articles.

Symptoms:
You may experience one or more the below symptoms.

• The Aria Operations UI login page displays:

Data retriever is not initialized yet.

• Navigating Aria Operations UI may display the error 

Unable to connect Platform Services

• The Aria Operations cluster is showing offline in Aria Operations Admin UI or it can show as online but individual node status is:
  
  Waiting for Analytics
  
  
• The Aria Operations Admin UI reports:

The Cluster was shut down because one node was out of disk space.

• The Aria Operations analytics logs /storage/log/vcops/log/analytics-<uuid>.log have messages similar to:

  ERROR [FsdbDataRetentionManager]  com.vmware.vcops.fsdb.FsdbDataRetentionManager.deleteResourceData - Delete resource data request is failed for resource #### :com.integrien.alive.FSDB.LowDiskSpaceException: Failed to increase file size. File /usr/lib/vmware-vcops/data/#/####/####_##_####.dat. Reason: FSDB is running low on disk space

• Aria Operations UI shows the following alert(s):
  • Disk space on node is low   
  • Fsdb is running critically low on disk space
  • Fsdb is estimated to run out of disk space soon
  • vRealize Operations Cluster database is running out of disk space

• Running df -h command on the Aria Operations nodes shows one or more file systems above 80% used.
• Accessing the product UI may throw the error :

ERR_TOO_MANY_REDIRECTS

 

Environment

VMware Aria Operations 8.x

Resolution

To verify which volume is out of space, run the df -h command on each Aria Operations node.  See the Related Information section for more information.

/storage/db is out of space

• Increase the size of the /storage/db partition: How to add disks to a node in Aria Operations cluster (318768)
  • Adding disk larger than 2TB: Disks larger than 2TB do not increase the /storage/db volume in VMware Aria Operations (342739)
• Clean up space in /storage/db partition: Heapdumps filling /storage/db in Aria Operations (317811)
• If the cluster was deployed through the vSphere Content Library: New appliance VM deployed from content library is not functioning (324369)

/storage/log is out of space

• Clean up logs from failed log rotation: /storage/log is full on Aria Operations (318019)
• Clean up space in /storage/log partition: How to safely clean /storage/log in Aria Operations (315972)
• Clean up excessive logging for local6: /storage/log partition full when Local6 logs are not rotated in VMware Aria Operations (314221)

/ is out of space

• Clean up space in the / partition: / is full on Aria Operations 8.x (310960)

Additional Information

To run the df -h command on a Aria Operations node, follow the steps below.

1. Log into the Primary node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
2. Run the following command:

df -h

3. You will see output including the below volumes:

Filesystem             Size  Used Avail Use% Mounted on
...
/dev/sda4               16G  4.5G   11G  30% /
...
/dev/mapper/data-db    207G   25G  172G  13% /storage/db
/dev/mapper/data-core   20G  2.3G   17G  92% /storage/core
/dev/mapper/data-log    20G  7.5G   12G  41% /storage/log
...

Impact/Risks:
To avoid cluster stability issues, do NOT expand an existing disk.