How to safely clean /storage/log in Aria Operations

Products

• VMware Aria Suite

Issue/Introduction

This article provides the safe procedure to reclaim /storage/log space consumed in Aria Operations.

 

Note: If this article does not resolve your issue, see Troubleshooting Storage Issues in Aria Operations (318408).

 

Possible Symptoms:

• Node inaccessible due to out of disk space.
• Unable to bring the cluster online because one or more nodes are out of disk space.

Environment

Aria Operations 8.x

Resolution

 

It is generally inadvisable to delete log files from a running system, as it can cause issues with file permissions, missing directories, and other unfulfilled expectations of the services writing them.  However, space can be reclaimed by marking the files as 0 bytes via the truncate command.

By combining truncate with the find command, you can ensure that only older log files are reduced in size to have their space reclaimed.

 

 

1. Log into the Aria Operations node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
2. Run the following command to truncate the logs:
   find /storage/log/ -mount -type f -mtime +1 -exec echo {} \; -exec truncate -cs 0 {} \; 2>&1 | tee /tmp/files_truncated.txt
3. Repeat steps 1-2 on all other affected nodes.

After running the truncation, it is advised to bring the cluster down and back up through the Admin UI to ensure that everything is working properly.

In case of any issues with bringing the cluster down or back up through the Admin UI please review the supported Shutdown and Startup sequence for a Aria Operations cluster (341964).

Additional Information

Here is a breakdown of the find options:

• /storage/log/
  • This is the data volume where all logs are kept.
• -mount
  • This prevents find from looking outside the log volume.
• -type f
  • This prevents find from looking at directories, links, or other special files.
• -mtime +1
  • This causes find to only look at files that have not been modified in the last 1 day. You can increase or remove this value as desired.
• -exec echo {} \;
  • This prints file paths that are discovered so that you can see affected files.
• -exec truncate -cs 0 {} \;
  • This truncates the file that is discovered to be 0 bytes.
• 2>&1 | tee /tmp/files_truncated.txt
  • This records all output of this command to a file at /tmp/files_truncated.txt

You can also add something similar to -iname *log* to the find command between -mtime and -exec to limit the search to only log files. However, this omits other files from being truncated such as heap dumps and gemfire replay data that generally take up more space than the log files do, if they exist.

 

Impact/Risks:

• Running the truncation command removes all history of your logs, which may contain useful information about other issues that may be present, and present a challenge to VMware support if you open a request with them.
• Logs only take up space on one of three data volumes. If you are having free space issues relating to one of the other volumes getting full, truncating log data does not help you.
• Do not run a truncation of files on any of the other data volumes, as it may cause irrecoverable damage to your system and loss of historical data.
• Because truncation leaves empty files behind, inodes in the filesystem remains consumed. This is not typically an issue, as most files are named in a manner that is subject to rolling out (and thus are finite in number), and because Aria Operations is tuned to allow significantly more inodes than most common Linux systems. If you find that after truncation and a full restart of the cluster services you have a stable system, you can use find to locate all 0-byte files and remove them. While relatively safe, this is not recommended unless you are actually running out of inodes. You can check this by running the command df -i.

It is recommended to take a snapshot following How to take a Snapshot of Aria Operations (342576) before proceeding with the steps below.