How to safely clean /storage/var space in Aria Operations for Logs (Formerly vRealize Log Insight)

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

This article provides a safe procedure to reclaim disk space consumed by logs in Aria Operations for Logs (formerly vRealize Log Insight.)

Environment

vRealize Log Insight 8.x

Aria Operations for Logs 8.x

Resolution

In general, it's safe to delete all the log files, but in the scope of this document, only log files older than one day are deleted.

1. Log into the Aria Operations for Logs node as root via SSH or Console, pressing ALT+F1 in a Console to log in.
2. Run the following command to truncate the logs:

find /storage/var/loginsight
-maxdepth 1 -mount -mtime +1 -type f -regex ".*[\._]log.+" -exec echo {} \; -exec rm {} \; 2>&1 | tee /tmp/deleted_logs.txt

3. Repeat steps 1-2 on all other affected nodes.

Note: To delete logs generated specifically by Apache Tomcat, execute the following command:

find apache-tomcat/logs -maxdepth 1 -mount -mtime +1 -type f -exec echo {} \; -exec rm {} \; 2>&1 | tee /tmp/deleted_tomcat_logs.txt

This command is similar to the command above with a slight deference. Apache Tomcat logs have format LogName.date.log and it's hard determine if it's an already rotated log file or the one currently in use. Thus, we are deleting all the files that has modification date greater than 1 day (the -mtime +1 option).
 

Additional Information

• /storage/var/loginsight
  • This is the data volume where all logs are kept.
    
    
• -maxdepth 1
  • Maximum number of nested directories to search through. (1 for only in current directory without going deeper)
    
    
• -mount
  • This prevents find from looking outside the log volume.
    
    
• -type f
  • This prevents find from looking at directories, links, or other special files.
    
    
• -mtime +1
  • This causes find to only look at files that have not been modified in the last 1 day. You can increase or remove this value as desired

• -regex ".*[\._]log.+" 
  • The regular expression to search pattern of log files.

• -exec echo {} \;
  • This prints file paths that are discovered so that you can see affected files.
    
    
• -exec rm {} \;
  • This removes the file 
    
    
• 2>&1 | tee /tmp/deleted_logs.txt
  • This records all output of this command to a file at /tmp/deleted_logs.txt

• find / -size +10M -exec du -h {} \; | sort -h
  • This records all the file equal and higher to 10M

Impact/Risks:

• 97% disk space usage on /storage/core is considered normal and healthy operation for Aria Operations for Logs.
• Logs in this document refer to the internal logs generated by Aria Operations for Logs, and containing information about its execution and not the logs that are ingested and stored within
• Logs generated by Aria Operations for Logs have a rotation period by default and usually should not occupy too much space. The rotation period is set from 5-10 days for most of the logs, but others do not have strict time rules. These can be bound to disk space (e.g. keep maximum of 10 log files with maximum 500MB disk usage).
• Running the following commands will delete all the logs generated by Aria Operations for Logs, which might contain useful information about other issues that may be occurring, and present a challenge to VMware support if you open a request. It would be a good idea to copy the logs off first if you are concerned with losing useful data.
  
  

Important: It is recommended to take a snapshot of the Aria Operations for Logs VM(s) before proceeding. Please remember to delete the snapshots within ~72 hours to prevent performance issues