Free disk space field reports high disk usage in the System Health view due to auth.log, messages and journalctl logs

Products

• VMware Cloud Director
• VMware Live Recovery

Issue/Introduction

• In the System Health view of the VMware Cloud Director Availability Provider Portal, the free disk space is low and displays a warning symbol in the Service status section.
• Running the df -h command shows that the (/) root or /opt partition is at or near 100% full.
• Log bundle creation fails.

Environment

VMware Cloud Director Availability 4.x

Cause

This issue is caused when the journalctl logs, /var/log/messages and /var/log/auth.log files increase and are not rotated or cleaned up. This results in a large amount of the space on the root filesystem being consumed.

Resolution

To remediate high disk utilization due to these files, confirm the top space-consuming files and directories within the affected filesystem and reduce their size as follows:

1. SSH to the impacted Cloud Director Availability appliance and log in as root.
2. Check the current disk space usage on the appliance:
   
   df -h
   
   
3. List the the top 10 files taking up space on the appliance:
   
   cd /
find -type f -exec du -Sh {} + | sort -rh | head -n 10
   
   Review the output and determine what are the top consumers of space.
   
   
4. Generally messages and auth.log files will be among the top files taking up space on the appliance. To blank out the messages and auth.log files to reclaim the used space perform these commands:
   
   echo > /var/log/messages
echo > /var/log/auth.log
   
   
5. Verify that the disk usage has been sufficiently reduced.
6. If there is still not enough free space, please run the following command to reduce the size of the journalctl logs:
   
   journalctl -–vacuum-size=200M 
   
   
7. Verify that the disk usage has been sufficiently reduced.
8. In a browser, navigate to the System Health view to confirm the disk space warning is gone.

Additional Information

For more information on other ways to free space, see Free up VMware Cloud Director Availability appliance disk space.