Troubleshooting Storage Issues in VCF Operations

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

This article provides troubleshooting steps for resolving storage capacity issues in VCF / Aria Operations.

You may be experiencing a storage issue if one or more file systems on your VCF / Aria Operations nodes exceed 80% utilization (verifiable by running the df -h command on each node).



UI Errors:

• Data retriever is not initialized yet

• Unable to connect Platform Services

• Waiting for Analytics

• The Cluster was shut down because one node was out of disk space

• ERR_TOO_MANY_REDIRECTS

• Disk space on this appliance is running low and filling up rapidly. If no action is taken immediately, services and UI access will go down. Add more disk space immediately to prevent service disruption

Active Alerts:

• Disk space on node is low

• Fsdb is running critically low on disk space (or is estimated to run out soon)

• vRealize Operations Cluster database is running out of disk space

Log Errors:

• You may see below exceptions in the /storage/log/vcops/log/analytics-<uuid>.log file. 

  ERROR [FsdbDataRetentionManager]  com.vmware.vcops.fsdb.FsdbDataRetentionManager.deleteResourceData - Delete resource data request is failed for resource #### :com.integrien.alive.FSDB.LowDiskSpaceException: Failed to increase file size. File /usr/lib/vmware-vcops/data/#/####/####_##_####.dat. Reason: FSDB is running low on disk space

  ERROR [Regular Data storage worker thread 1]  com.vmware.vcops.fsdb.datareceiver.StorageWorkItem.saveObservation - Save metric data failed: Failed to increase file size. File /usr/lib/vmware-vcops/data/#/##/####_##_##.dat. Reason: FSDB is running low on disk space

Environment

VMware Aria Operations 8.18.x

VMware Cloud Foundation Operations 9.0.x

Resolution

Proceed with the relevant solution below:

1. /storage/db is out of space
   • Increase the size of the /storage/db partition: How to add disks to a node in Aria Operations cluster
     • Adding disk larger than 2TB: Disks larger than 2TB do not increase the /storage/db volume in VMware Aria Operations
   • Clean up space in /storage/db partition: Heapdumps filling /storage/db in Aria Operations
   • If the cluster was deployed through the vSphere Content Library: New appliance VM deployed from content library is not functioning
     
     
2. /storage/log is out of space
   
   • Clean up logs from failed log rotation: /storage/log is full on Aria Operations
   • Clean up space in /storage/log partition: How to safely clean /storage/log in Aria Operations
   • Clean up excessive logging for local6: /storage/log partition full when Local6 logs are not rotated in VMware Aria Operations
   
   
3. / is out of space
   • Clean up space in the / partition: / is full on Aria Operations 8.x

Note : Expanding an existing disk is not supported and may result in cluster instability.