Troubleshooting Storage Issues in Aria Operations

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• UI errors

Data retriever is not initialized yet

Unable to connect Platform Services

Waiting for Analytics

The Cluster was shut down because one node was out of disk space.

ERR_TOO_MANY_REDIRECTS

• Active alert(s)
  • Disk space on node is low   
  • Fsdb is running critically low on disk space
  • Fsdb is estimated to run out of disk space soon
  • vRealize Operations Cluster database is running out of disk space

• Logs error
  
  /storage/log/vcops/log/analytics-<uuid>.log :
  ERROR [FsdbDataRetentionManager]  com.vmware.vcops.fsdb.FsdbDataRetentionManager.deleteResourceData - Delete resource data request is failed for resource #### :com.integrien.alive.FSDB.LowDiskSpaceException: Failed to increase file size. File /usr/lib/vmware-vcops/data/#/####/####_##_####.dat. Reason: FSDB is running low on disk space
  
  

• Running df -h command on the Aria Operations nodes shows one or more file systems above 80% used. To verify which volume is out of space, run the df -h command on each Aria Operations node.
  
  
  1. Log into the Primary/Data node as a root via SSH or Console, pressing ALT+F1 in a Console to log in.
  2. Run the following command:
  df -h
  
  
  

Environment

VMware Aria Operations 8.x

VMware Cloud Foundation Operations 9.0.x

Resolution

Proceed with the relevant solution below:

1. /storage/db is out of space
   • Increase the size of the /storage/db partition: How to add disks to a node in Aria Operations cluster
     • Adding disk larger than 2TB: Disks larger than 2TB do not increase the /storage/db volume in VMware Aria Operations
   • Clean up space in /storage/db partition: Heapdumps filling /storage/db in Aria Operations
   • If the cluster was deployed through the vSphere Content Library: New appliance VM deployed from content library is not functioning
     
     
2. /storage/log is out of space
   
   • Clean up logs from failed log rotation: /storage/log is full on Aria Operations
   • Clean up space in /storage/log partition: How to safely clean /storage/log in Aria Operations
   • Clean up excessive logging for local6: /storage/log partition full when Local6 logs are not rotated in VMware Aria Operations
   
   
3. / is out of space
   • Clean up space in the / partition: / is full on Aria Operations 8.x
     
     

Note : Expanding an existing disk is not supported and may result in cluster instability.