Aria Operations cluster is down due to "/storage/log" partition full at 100% disk usage

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

• Aria Operation cluster is down. 
• UI is not accessible.
• UI is showing "Unable to connect to platform services"

Environment

Aria Operations 8.x

Cause

/storage/log partition is at 100%.  When this log partition is full, services cannot write log to the partition, causing these services to stop running.

Resolution

SSH as root into the Aria Operations node.

Perform the following command to safely cleanup the log from the log partition:  find /storage/log/ -mount -type f -mtime +1 -exec echo {} \; -exec truncate -cs 0 {} \; 2>&1 | tee /tmp/files_truncated.txt

Once the storage issue has been resolved via the cleaning of the log partition, proceed to take cluster offline.

Login to the Aria Operations admin UI.  Click on "Take Cluster Offline".

Once cluster is Offline, click on "Take Cluster Online" to bring cluster back Online.

Additional Information

Reference KB How to safely clean /storage/log in Aria Operations

Reference KB Troubleshooting Storage Issues in Aria Operations