/dev/sda3 - root partition 100% full due to Audit.log files not being rotated in vCenter Server Appliance

Products

• VMware vCenter Server

Issue/Introduction

This article provides steps to reduce the audit.log size.

Symptoms:

• 100% capacity used for /dev/sda3.
• Size of audit.log file is very large and /var/log/audit folder consumes majority of the space.
• Saved logs from log rotate policy reference a date that is not in line with the policy.
• Unable to connect to the vCenter Server as services are not started.
• Running /etc/cron.daily/logrotate manually rotates logs as expected.
• Accessing vSphere Web Client might fail with error: 503 service unavailable

Environment

VMware vCenter Server Appliance 6.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 5.5.x

Resolution

To resolve this issue, truncate the audit.log file and verify the cron job is working correctly.
 

Truncate audit.log

1. Log in to the vCenter Server Appliance through SSH.
2. Run this command to enable access the Bash shell:
   
   shell.set --enabled true
    
3. Type shell and press Enter.
4. Navigate to the /var/log/audit folder with this command:
   
   cd /var/log/audit
   For 6.7:

cd /var/spool/anacron/

1. Run this command to verify the issue is with the audit.log file being too large (a few GBs):
   
   ls -lh
   
   For example:
   
   ls -lh
   
   total 3.5G
   -rw------- 1 root root 3.5G Feb 3 16:55 audit.log
   -rw------- 1 root root 445K Apr 8 2016 audit.log-20160408.bz2
   -rw------- 1 root root 447K Apr 9 2016 audit.log-20160409.bz2
    
2. Truncate (clean the content without deleting the file) the audit.log file with this command:

truncate -s 0 audit.log
 

Verify that the cron job to rotate the audit.log is running

1. Run this command to see when the cron job was last ran successfully:

ls -l /var/spool/cron/lastrun/

For example:

ls -l /var/spool/cron/lastrun/

total 0
-rw------- 1 root root 0 Apr 22 2016 cron.daily
-rw------- 1 root root 0 Apr 22 2016 cron.hourly
-rw------- 1 root root 0 Apr 21 2016 cron.weekly

 

2. Determine if the cron job was last updated long time ago. Normally, this should be daily.
3. Run this command to check for credential failures running the cron job:

 

grep "Authentication token is no longer valid; new one required" /var/log/messages.0.log | head

 

For example:

grep "Authentication token is no longer valid; new one required" /var/log/messages.0.log | head

2016-11-07T00:20:01.617180+00:00 vcenter /usr/sbin/cron[18972]: Authentication token is no longer valid; new one required
2016-11-07T00:20:01.617183+00:00 vcenter /usr/sbin/cron[18974]: Authentication token is no longer valid; new one required

 

4. Run this command to check if the root password has expired:

 

chage -l root

For example:

chage -l root

Password change requested. Choose a new password.
Old Password:
New password:

 

5. Change the root password as prompted.
6. Verify the root account password has been changed:

chage -l root

For example:

chage -l root

Minimum: 0
Maximum: 365
Warning: 7
Inactive: -1
Last Change: Feb 03, 2017
Password Expires: Feb 03, 2018
Password Inactive: Never
Account Expires: Never

7.  Restart all vCenter Server services.

service-control --stop --all
service-control --start --all

Note: Run the below command to change the root password to never expire:

#chage -m 0 -M 99999 -I -1 -E -1 root

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
vCenter Server Appliance Maintenance Series: Troubleshooting disk space related issues
由于 Audit.log 文件未进行轮换，vCenter Appliance root 分区已满
EAM Service fails to start after vCenter Server reboot
/storage/log directory is full in vCenter Server Appliance 6.0