SDDC Manager UI becomes inaccessible due to root (/dev/sda4) partition filling up

Products

• VMware Cloud Foundation
• VMware SDDC Manager / VCF Installer

Issue/Introduction

• SDDC Manager UI may be inaccessible with error: 503 Service Unavailable
• The user interface may fails to load and remains indefinitely stuck on the splash screen message: VMware Cloud Foundation is initializing...
  
  

Environment

• VCF 5.x/9.x
• SDDC 5.x/9.x

Cause

• Root partition on SDDC is full or almost full.
• Checking directories for large with command df -h or du -sh /var/log/* | sort -nr, results in below output.

         

• When the root filesystem reaches maximum capacity, critical SDDC Manager microservices, internal databases, and underlying logging frameworks lose the ability to write temporary files, PID files, or runtime data. This storage exhaustion blocks essential orchestration services (such as vcf-commonsvcs or operationsmanager) from completing their startup sequences, locking the UI into a perpetual initialization loop.
• Log rotation is not occurring on the SDDC Manager appliance due to misconfigurations with the log rotation system files.

Resolution

To resolve this issue, follow the steps below:

Note: Take Snapshot of the SDDC Manager.

1. SSH into SDDC Manager, login as  vcf user, login to root with the command su root.
2. cd to /var/log/ directory and find out the maximum sized log files and delete them.
   
   
   • Note: If the root partition is still full or the /var/log/messages log is over 1GB, manually clear the messages logs and auth.log prior to running above steps.

     echo > /var/log/messages
echo > /var/log/audit/audit.log
echo > /var/log/auth.log
echo > /var/log/nginx/error.log
echo > /var/log/nginx/access.log

3. If the above does not clear out the space you can delete all the compressed log files as well, using following commands:
   
   rm /var/log/audit/audit.*.gz
rm /var/log/messages.*.gz
rm /var/log/auth.*.gz
rm /var/log/nginx/error.*.gz
rm /var/log/nginx/access.*.gz

 

To prevent the issue further, follow the steps below:

1. SSH into SDDC Manager, login as  vcf user, login to root with the command su root
2. Verify that the /etc/logrotate.conf file is present, and contains a line exactly as follows: include /etc/logrotate.d
3. Navigate to the logrotate.d directory.
   
   Note: Take a backup of existing syslog file before copying the file from this KB into the SDDC Manager.
   
   cd /etc/logrotate.d
   
   
4. Using WinSCP, copy the syslog file from this KB to the /etc/logrotate.d directory.
   
   

   Note: While copying the file to /etc/logrotate.d the following error may arise:  "Permission denied". 

              

        To fix this issue, use any of the methods mentioned below:

   Method 1: Copy the syslog file mention in the KB to the /vcf/home/ directory and then, move the file to /etc/logrotate/ directory, using following command:
   
   mv syslog /etc/logrotate.d/
   

   Method 2: Copy the content of the of syslog file and by using the vi editor, paste the content into the current syslog file. If there isn't a syslog file in the directory, create a new file using the vi editor (vi syslog).

5. Update the permissions on the file, using following command:
   
   chown root:root syslog
chmod 644 syslog
   
   
6. Initiate the log rotation, using following command:
   
    logrotate -f /etc/logrotate.conf

Additional Information

• If the log rotate fails with below error: 
  

  error: destination /var/log/audit/audit.log-####.gz already exists, skipping rotation
error: destination /var/log/nginx/access.log-####.gz already exists, skipping rotation
error: destination /var/log/nginx/error.log-####.gz already exists, skipping rotation

  Then manually remove the above .gz files from the respective directory and re-run the log rotate Task, as mentioned in Point 6.

• Additionally, the Max_File_size.sh script can be used to check the root partition and see exactly which files and folders are consuming the most space.
  

  Steps to perform:

  Take the snapshot of the sddc manager 

  • login to /home/vcf
  • vi cleanup.sh
  • Paste the script and save the file 
  • Give permission to the file : chmod +x cleanup.sh
  • Run the command : ./Max_File_size.sh