SDDC Manager UI becomes inaccessible due to root (/dev/sda4) partition filling up

Products

• VMware Cloud Foundation

Issue/Introduction

• SDDC Manager UI may be inaccessible with 503 Service Unavailable errors.

• Root partition on SDDC is full or almost full. 

         

• auth.log and/or message log are not rotating and filling up the root partition

• vg_system-lv_root is full 

 

Environment

VMware Cloud Foundation 5.x

Cause

Log rotation is not occurring on the SDDC Manager appliance due to misconfigurations with the log rotation system files.

Resolution

Important: If the root partition is full or messages log is over 1GB you should manually clear the messages log and auth.log prior to running these steps.
echo > /var/log/messages
echo > /var/log/audit/audit.log
echo > /var/log/auth.log

 

To resolve the issue:

1. SSH into SDDC Manager, login as  vcf user, login to root with the command su root
2. Verify that the /etc/logrotate.conf file is present, and contains a line exactly as follows:    include /etc/logroate.d
3. Navigate to the logrotate.d directory. Note: any files in this directory will be read when the logrotations take place.
   cd /etc/logrotate.d
   
   
4. By using WinSCP, copy the syslog file from the KB to the /etc/logrotate.d directory.

Note: - While copying the file to /etc/logrotate.d error may come as "Permission denied". 

           

     To fix this issue use any of the below mentioned methods:

Method #1 Send the syslog file mention in the KB to /vcf/home directory then move the file to /etc/logrotate.d

               mv syslog /etc/logrotate.d

Method #2 Copy the content of the of syslog file and by using vi editor paste the content in the current syslog file 

 

      5. Update the permissions on the file.
     chown root:root syslog
     chmod 644 syslog

      6. Initiate the log rotation 
logrotate -f /etc/logrotate.conf

 

Additional Information

While running the command logrotate -f /etc/logrotate.conf if error came as: - 

error: destination /var/log/audit/audit.log-20241203.gz already exists, skipping rotation
error: destination /var/log/nginx/access.log-20241203.gz already exists, skipping rotation
error: destination /var/log/nginx/error.log-20241203.gz already exists, skipping rotation

To resolve the issue manually removed the files from the directory then again try running logrotate -f /etc/logrotate.conf