SDDC Manager UI becomes inaccessible due to root (/dev/sda4) partition filling up

Products

• VMware Cloud Foundation

Issue/Introduction

• SDDC Manager UI may be inaccessible with 503 Service Unavailable errors.

• Root partition on SDDC is full or almost full. 

         

• auth.log and/or message log are not rotating and filling up the root partition

• vg_system-lv_root is full 

 

Environment

VMware Cloud Foundation

Cause

The required log rotation file is missing from SDDC Manager appliance.

Resolution

To resolve the issue

1. SSH into SDDC Manager, login as  vcf user, login to root with the command su root
2. Navigate to the logrotate.d directory
   cd /etc/logrotate.d
   
   
3. By using WinSCP, copy the syslog file from the KB to the directory.

Note: - While copying the file to /etc/logrotate.d error may come as "Permission denied". 

           

     To fix this issue use any of the below mentioned method 

Option #1 Send the syslog file mention in the KB to /vcf/home directory then move the file to /etc/logrotate.d

               mv syslog /etc/logrotate.d

Option#2 Copy the content of the of syslog file and by using vi editor paste the content in the current syslog file 

      4. Update the permissions on the file.
     chown root:root syslog
     chmod 644 syslog

      5. Initiate the log rotation 
logrotate -f /etc/logrotate.conf

Important: If the root partition is full or messages log is over 1GB you should manually clear the messages log and auth.log prior to running this workaround.
echo > /var/log/messages
echo > /var/log/audit/audit.log
echo > /var/log/auth.log

Additional Information

While running the command logrotate -f /etc/logrotate.conf if error came as: - 

error: destination /var/log/audit/audit.log-20241203.gz already exists, skipping rotation
error: destination /var/log/nginx/access.log-20241203.gz already exists, skipping rotation
error: destination /var/log/nginx/error.log-20241203.gz already exists, skipping rotation

To resolve the issue manually removed the files from the directory then again try running logrotate -f /etc/logrotate.conf