How to safely clean /storage/log in Aria Operations

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

This article provides the safe procedure to reclaim /storage/log space consumed in Aria Operations.

 

Note: If this article does not resolve your issue, see Troubleshooting Storage Issues in VCF Operations (318408).

 

Possible Symptoms:

• Node inaccessible due to out of disk space.
• Unable to bring the cluster online because one or more nodes are out of disk space.
• Running df -h on Aria Operations node root ssh session shows /storage/log partition has reached above 95%
• Product UI is accessible but dashboards are showing no results

When attempting to access the browser UI, the following message is displayed:
“vRealize Operations Manager is not available. You will be redirected when the service is available.”

Environment

Aria Operations 8.x

Cause

The appliance's native log rotation mechanism (e.g., logrotate) fails to compress or clear older logs, allowing them to grow indefinitely.

Specific services within Aria Operations (like Cassandra, Gemfire, or the casa service) enter an error loop, rapidly spamming logs (such as heap dumps or replay data) faster than the system can clear them.

When the partition fills up, active services maintain open file locks on the log files. Because the files are actively locked by running processes, simply expanding the VMDK or using standard rm commands will not release the storage blocks back to the file system. The truncate method is required to empty the files while respecting the active file locks.

Resolution

 

It is generally inadvisable to delete log files from a running system, as it can cause issues with file permissions, missing directories, and other unfulfilled expectations of the services writing them. However, space can be reclaimed by marking the files as 0 bytes via the truncate command.

By combining truncate with the find command, you can ensure that only older log files are reduced in size to have their space reclaimed.

 

It is recommended to take a snapshot following Snapshot Creation in VMware Aria Operations(342576) before proceeding with the steps below.

1. Log into the Aria Operations node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
2. Run the following command to truncate the logs (Review Additional information section below before running the command):
   find /storage/log/ -mount -type f -mtime +1 -exec echo {} \; -exec truncate -cs 0 {} \; 2>&1 | tee /tmp/files_truncated.txt
3. Repeat steps 1-2 on all other affected nodes.

After running the truncation, it is advised to bring the cluster down and back up through the Admin UI to ensure that everything is working properly.

In case of any issues with bringing the cluster down or back up through the Admin UI please review the supported Shutdown and Startup sequence for VCF/Aria Operations cluster (341964).

Additional Information

Here is a breakdown of the find options:

• /storage/log/
  • This is the data volume where all logs are kept.
• -mount
  • This prevents find from looking outside the log volume.
• -type f
  • This prevents find from looking at directories, links, or other special files.
• -mtime +1
  • This causes find to only look at files that have not been modified in the last 1 day. You can increase or remove this value as desired.
• -exec echo {} \;
  • This prints file paths that are discovered so that you can see affected files.
• -exec truncate -cs 0 {} \;
  • This truncates the file that is discovered to be 0 bytes.
• 2>&1 | tee /tmp/files_truncated.txt
  • This records all output of this command to a file at /tmp/files_truncated.txt

You can also add something similar to -iname *log* to the find command between -mtime and -exec to limit the search to only log files. However, this omits other files from being truncated such as heap dumps and gemfire replay data that generally take up more space than the log files do, if they exist.

 

Impact/Risks:

• Running the truncation command removes all history of your logs, which may contain useful information about other issues that may be present, and present a challenge to VMware support if you open a request with them.
• Logs only take up space on one of three data volumes. If you are having free space issues relating to one of the other volumes getting full, truncating log data does not help you.
• Do not run a truncation of files on any of the other data volumes, as it may cause irrecoverable damage to your system and loss of historical data.
• Because truncation leaves empty files behind, inodes in the filesystem remains consumed. This is not typically an issue, as most files are named in a manner that is subject to rolling out (and thus are finite in number), and because Aria Operations is tuned to allow significantly more inodes than most common Linux systems. If you find that after truncation and a full restart of the cluster services you have a stable system, you can use find to locate all 0-byte files and remove them. While relatively safe, this is not recommended unless you are actually running out of inodes. You can check this by running the command df -i.

Supplementary KB Context

• KB 318019: Addresses a specific, known Photon OS bug where system logs (/var/log/warn, /var/log/auth.log, /var/log/messages) bypass log rotation and aggressively fill the disk. Used to identify if the OS itself (rather than an Aria service) is the culprit.

• KB 318408: The master storage troubleshooting guide. It provides the overarching threshold rules (alerts trigger at >80%, critical failures at >95%) and dictates the post-cleanup procedures, such as restarting the cluster to ensure all nodes properly register the newly freed capacity.