/storage/log partition full when Local6 logs are not rotated in VMware Aria Operations

Products

• VMware Aria Suite

Issue/Introduction

The purpose of this KB is to provide a list of steps to verify whether logs fail to rotate and how to resolve the issue.

Local6 logs are not rotated in VMware Aria Operations (Formerly known as VMware vRealize Operations Manager) due to a directory permission.

Environment

VMware vRealize Operations 8.10.x
VMware Aria Operations 8.12.x

Cause

• Log file /var/log/local6 fails to rotate due to incorrect owner and group of parent directories.
• Directories /storage/log/var and /storage/log/var/log has wrong owner and group as per example below:
  

  drwxrwxr-x 13 admin admin 4.0K May 24 14:53 log

Resolution

This issue has been resolved in Aria Operations 8.12 HF1.

 

Workaround:

chown root:root /storage/log/var
chown root:root /storage/log/var/log
/usr/sbin/logrotate --verbose /etc/logrotate.conf

 

As an emergency procedure when root filesystem is full, you can truncate the file to 0 bytes to free up space used by the log file:

truncate -s 0 /var/log/local6

Additional Information

The owner and group of /storage/log/var and /storage/log/var/log should be 'root' and 'root', as per sample output below:

root@vrops01 [ /storage/log ]# ls -al
total 32
drwxrwxr-x 5 root  root   4096 Jul 16 16:44 .
drwxrwxr-x 8 root  root   4096 Aug 10 09:25 ..
drwxrwxr-x 2 admin admin 16384 Jul 16 16:38 lost+found
drwxrwxr-x 3 root  root   4096 Jul 16 16:43 var
drwxrwxr-x 4 admin admin  4096 Jul 16 16:45 vcops

root@vrops01 [ /storage/log/var ]# ls -al
total 12
drwxrwxr-x  3 root root 4096 Jul 16 16:43 .
drwxrwxr-x  5 root root 4096 Jul 16 16:44 ..
drwxrwxr-x 13 root root 4096 Sep 25 03:30 log