Log Rotation not working on SDDC Manager.

Products

• VMware Cloud Foundation

Issue/Introduction

There is an error in the SDDC Manager UI when logging in as the same SSO user : 

status 500 error

And while changing the password of VCF getting error : "Failed to start Postgres".

From the SDDC VM console login with root credential and check the partition :

df -h

Environment

VMware Cloud Foundation 5.x

Cause

•  Log rotation is not occurring on the SDDC Manager appliance due to misconfigurations with the log rotation system files.

Resolution

Note : Please take the snapshot of SDDC VM before start the activity.

To resolve the issue,  manually clear the messages log and auth.log files:

echo > /var/log/messages
echo > /var/log/audit/audit.log
echo > /var/log/auth.log

You can run the following to help with residual "big files" that might be filling up the root partition:

du -ah ./ | grep '[0-9][GM]' | head -n 20 | sort -r -n

If this command returns any files that are not in syslog or logrotate.conf, it means that those files aren't getting rotated and compressed.  For example: /var/log/messages.1 is several gigabytes large, 

In this situation, File messages.1 would need to be clear manually which should automatically start the UI and SSH as well.

 

To resolve the auto log rotation issue:

1. SSH into SDDC Manager, login as  vcf user, login to root with the command su root
2. Verify that the /etc/logrotate.conf file is present.

Update the permissions of syslog under /etc/logrotate.d :
     chown root:root syslog
     chmod 644 syslog

Initiate the log rotation 
logrotate -f /etc/logrotate.conf

 

While running the command logrotate -f /etc/logrotate.conf if error came as: - 

error: destination /var/log/audit/audit.log-20241203.gz already exists, skipping rotation
error: destination /var/log/nginx/access.log-20241203.gz already exists, skipping rotation
error: destination /var/log/nginx/error.log-20241203.gz already exists, skipping rotation

To resolve the issue, manually remove the above files from the directory then again try running logrotate -f /etc/logrotate.conf.