SDDC Manager UI becomes inaccessible due to root (/dev/sda4) partition filling up

Products

• VMware Cloud Foundation

Issue/Introduction

• SDDC Manager UI may be inaccessible with 503 Service Unavailable errors.

• Root partition on SDDC is full or almost full. 

• Checking directories for large with command df -h or du -sh /var/log/* | sort -nr

         

• auth.log and/or message log are not rotating and filling up the root partition.

• The vg_system-lv_root partition is full. 

 

Environment

VMware Cloud Foundation 5.x

Cause

Log rotation is not occurring on the SDDC Manager appliance due to misconfigurations with the log rotation system files.

Resolution

Important: If the root partition is full or the /var/log/messages log is over 1GB you should manually clear the messages log and auth.log prior to running these steps.

echo > /var/log/messages
echo > /var/log/audit/audit.log
echo > /var/log/auth.log
echo > /var/log/nginx/error.log
echo > /var/log/nginx/access.log

If the above does not clear out the space you can delete all the compressed log files as well:

rm /var/log/audit/audit.*.gz
rm /var/log/messages.*.gz
rm /var/log/auth.*.gz
rm /var/log/nginx/error.*.gz
rm /var/log/nginx/access.*.gz

To prevent the issue from happening again:

1. SSH into SDDC Manager, login as  vcf user, login to root with the command su root
2. Verify that the /etc/logrotate.conf file is present, and contains a line exactly as follows:    include /etc/logrotate.d
3. Navigate to the logrotate.d directory. Note: any files in this directory will be read when the log rotations take place so if taking a backup of the current syslog file in this directory example syslog.old move it to another directory, otherwise it will clash with the main syslog file.

   cd /etc/logrotate.d

   
   
4. Using WinSCP, copy the syslog file from this KB to the /etc/logrotate.d directory.

Note: - While copying the file to /etc/logrotate.d the following error may be present:  "Permission denied". 

           

     To fix this issue, use any of the methods mentioned below:

Method #1 Send the syslog file mention in the KB to the /vcf/home/ directory. Then, move the file to /etc/logrotate/. 

mv syslog /etc/logrotate.d/

Method #2 Copy the content of the of syslog file and by using the vi editor, paste the content into the current syslog file. If there isn't a syslog file in the directory, create a new file using the vi editor (vi syslog).

 

      5. Update the permissions on the file. 

chown root:root syslog
chmod 644 syslog

 

      6. Initiate the log rotation. 

logrotate -f /etc/logrotate.conf

 

Additional Information

While running the command logrotate -f /etc/logrotate.conf if error came as: - 

error: destination /var/log/audit/audit.log-20241203.gz already exists, skipping rotation
error: destination /var/log/nginx/access.log-20241203.gz already exists, skipping rotation
error: destination /var/log/nginx/error.log-20241203.gz already exists, skipping rotation

To resolve the issue manually removed the files from the directory then again try running logrotate -f /etc/logrotate.conf