Large File Sizes of /var/log/messages and /var/mail/root Causing Root Partition to Fill Up in Aria Operations for Logs

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

This article provides steps to reduce the size of /var/log/messages and /var/mail/root

• You are unable to access Aria Operations for Logs user interface
• You notice high usage on /dev/sda-x (where x is typically 4 or 5).

Note: Run this command to check disk space:

df -h


Note: Run this command to check the largest files:

du -hcx / 2>/dev/null | sort -h -r | head -n 30

 

Note: Run this command to observe whether the messages log file size is large

ls -lh /var/log/messages
ls -lh /var/mail/root

 

See also reference The Aria Operations for Logs (Formerly vRealize Log Insight) root partition is full

If java_pid####.hprof files are present on a node please see Aria Operation for Logs shows disconnected and has 100% utilization on root partition

Environment

VMware Aria Operations for Logs 8.x

Cause

The file /var/mail/root is the default location for storing mail messages sent to the root user on Unix-like systems, such as Linux. This is typically the system mail file, where notifications, alerts, or error messages related to system operations, such as cron jobs or system processes, are sent to the root user.

Resolution

1. Create a snapshot without memory on all nodes in the Aria Operations for Logs cluster
2. SSH to the primary node using root credentials
3. Stop the Aria Operations for Logs service with the below command
    service loginsight stop
4. Run the below command to clear the content of the messages file
   Example: truncate -s 0 /var/log/messages
   Example: truncate -s 0 /var/mail/root

5. Reboot the node to ensure that the used space is cleared out

6. Repeat steps 2-5 on all remaining nodes in the cluster