/storage/log is full on Aria Operations

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Symptoms:

• The /storage/log partition is full on one or more Aria Operations node and these log files are consuming this space:
  • /var/log/auth.log
  • /var/log/messages
  • /var/log/warn

• When attempting to access the browser UI, the following message is displayed:
  “vRealize Operations Manager is not available. You will be redirected when the service is available.”

Environment

Aria Operations 8.x

 

Cause

In Aria Operations 8.x this is caused by the move to Photon OS and an incorrect log rotation mechanism.

Resolution

To resolve this issue remove the /var/log/warn, /var/log/auth.log, and /var/log/messages files.

1. Log into the Aria Operations node as root via SSH or Console, pressing ALT+F1 in a Console to log in.  
2. Run the following command to stop the syslog service:
   
   service syslog stop
   
   
3. Remove the /var/log/warn, /var/log/auth.log, and /var/log/messages files by running this command:
   
   rm -f /var/log/warn* /var/log/auth.log* /var/log/messages*
   
   
4. Run the following command to start the syslog service:
   
   service syslog start
   
   
5. Repeat steps 1-4 on any other affected nodes.

 

Note: In some scenarios, the /storage/log/var/log/httpd/access.log file will consume excessive disk space on the Analytics nodes. Verify the disk usage of the /storage/log/var/log/httpd/access.log file on the nodes using du -sh * command. If excessive usage is found, the truncate -s 0 /storage/log/var/log/httpd/access.log command can be used to immediately reclaim space.

Note: If this article does not resolve your issue:

• see Troubleshooting Storage Issues in Aria Operations (318408)
  
  
• If running 8.12.x there is a known issue where the /storage/log partition full when Local6 logs are not rotated in VMware Aria Operations

 

Additional Information

Impact/Risks:
Logs only fill the /storage/log partition. Removing log data will not reduce space in the other partitions.

It is recommended to take a snapshot following Snapshot Creation in VMware Aria Operations before proceeding with the steps below.