SDDC Manager UI becomes inaccessible due to root (/dev/sda4) partition filling up

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• SDDC Manager UI may be inaccessible with 503 Service Unavailable errors.

• Root partition on SDDC is full or almost full. 

• auth.log and/or message log are not rotating and filling up the root partition

• vg_system-lv_root is full 

 

Environment

VMware Cloud Foundation

Cause

The required log rotation file is missing from SDDC Manager appliance.

Resolution

 

Important: If the root partition is full or messages log is over 1GB you should manually clear the messages log and auth.log prior to running this workaround.
echo > /var/log/messages
echo > /var/log/audit/audit.log
echo > /var/log/auth.log

1. SSH into the SDDC Manager and change to root
2. Navigate to the logrotate.d directory
   cd /etc/logrotate.d
   
   
3. Copy the syslog file from the KB to the directory.
4. Update the permissions on the file.
   chown root:root syslog
   chmod 644 syslog
   
5.  Initiate the log rotation 
   logrotate -f /etc/logrotate.conf