Troubleshooting Storage Issues in Aria Operations

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Various storage related issues may arise in Aria Operations.
• To assist in troubleshooting, see the below list of the most common storage related articles.
• The Aria Operations UI login page displays:

Data retriever is not initialized yet.

• Navigating Aria Operations UI may display the error 

Unable to connect Platform Services

• The Aria Operations cluster is showing offline in Aria Operations Admin UI or it can show as online but individual node status is:
  
  Waiting for Analytics
  
  
• The Aria Operations Admin UI reports:

The Cluster was shut down because one node was out of disk space.

• The Aria Operations analytics logs /storage/log/vcops/log/analytics-<uuid>.log have messages similar to:

  ERROR [FsdbDataRetentionManager]  com.vmware.vcops.fsdb.FsdbDataRetentionManager.deleteResourceData - Delete resource data request is failed for resource #### :com.integrien.alive.FSDB.LowDiskSpaceException: Failed to increase file size. File /usr/lib/vmware-vcops/data/#/####/####_##_####.dat. Reason: FSDB is running low on disk space
  
  
• Aria Operations UI shows the following alert(s):
  • Disk space on node is low   
  • Fsdb is running critically low on disk space
  • Fsdb is estimated to run out of disk space soon
  • vRealize Operations Cluster database is running out of disk space

• Accessing the product UI may throw the error :

ERR_TOO_MANY_REDIRECTS

• Running df -h command on the Aria Operations nodes shows one or more file systems above 80% used. To verify which volume is out of space, run the df -h command on each Aria Operations node.
  
  
  1. Log into the Primary/Data node as a root via SSH or Console, pressing ALT+F1 in a Console to log in.
  2. Run the following command:
  df -h
  
  
  

Environment

VMware Aria Operations 8.x

Resolution

Proceed with the relevant solution below:

1. /storage/db is out of space
   • Increase the size of the /storage/db partition: How to add disks to a node in Aria Operations cluster
     • Adding disk larger than 2TB: Disks larger than 2TB do not increase the /storage/db volume in VMware Aria Operations
   • Clean up space in /storage/db partition: Heapdumps filling /storage/db in Aria Operations
   • If the cluster was deployed through the vSphere Content Library: New appliance VM deployed from content library is not functioning
     
     
2. /storage/log is out of space
   
   • Clean up logs from failed log rotation: /storage/log is full on Aria Operations
   • Clean up space in /storage/log partition: How to safely clean /storage/log in Aria Operations
   • Clean up excessive logging for local6: /storage/log partition full when Local6 logs are not rotated in VMware Aria Operations
   
   
3. / is out of space
   • Clean up space in the / partition: / is full on Aria Operations 8.x
     
     

Note : Expanding an existing disk is not supported and may result in cluster instability.