/storage/log partition full due to SSO log files are not compressed in vCenter Server Appliance

Products

VMware vCenter Server

Issue/Introduction

To work around the /storage/log directory full issue in vCenter Server Appliance, decrease the maximum backup size and the maximum backup index in the log4j.properties file.

Symptoms:
You see that the /storage/log directory is highly utilized in an Embedded vCenter Server Appliance or Platform Controller Server Appliance 6.0.

Note: For additional symptoms and log entries, see the Additional Information section.

Environment

VMware vCenter Server Appliance 6.5.x
VMware vCenter Server Appliance 6.0.x

Cause

This issue occurs due to an increase in the SSO log files in the /storage/log directory. During the log rotation, the old log file is not compressed resulting in multiple large files in the /storage/log directory.

Resolution

This issue is resolved in VMware vCenter Server 6.0 Update 3, available at VMware Downloads
The issue is also resolved in 6.5 Update 1, available at VMware Downloads

Workaround:

Video tutorial to Resolve/Workaround the issue:

To work around this issue, decrease the maximum backup size and the maximum backup index in the log4j.properties file.

Note: In external Platform Service Controller (PSC) environments, log in to the Platform Service Controller instead.
Note: This workaround is applicable only for 6.0 version only.

Process to decrease maximum backup size and index in the log4j.properties file:

Log in to the vCenter Server Appliance or PSC console as root user.
Run this command to enable access the Bash shell:

shell.set --enabled true
Type shell and press Enter.
Navigate to this directory:

cd /usr/lib/vmware-sso/vmware-sts/webapps/sts/WEB-INF/classes/
Run this command to back up the log4j.properties file:

cp log4j.properties log4j.properties.bak
Open the log4j.properties file using a text editor.
Locate the log4j.appender.LOGFILE.MaxFileSize and log4j.appender.LOGFILE.MaxBackupIndex parameters after this entry:

log4j.appender.LOGFILE.File=${catalina.base}/logs/vmware-identity-sts.log
Change the value of parameters to 50 and 5 respectively:

For example:

log4j.appender.LOGFILE.MaxFileSize=50MB
log4j.appender.LOGFILE.MaxBackupIndex=5

Note: The default values are MaxFileSize=100MB and MaxBackupIndex=10.
Locate the log4j.appender.PERFLOG.MaxBackupIndex parameter after this entry:

log4j.appender.PERFLOG.File=${catalina.base}/logs/vmware-identity-sts-perf.log
Change the value of parameter to 3.

For example:

log4j.appender.PERFLOG.MaxBackupIndex=3

Note: The default value is MaxBackupIndex=10.
Navigate to: cd /storage/log/vmware/sso/

Remove the old localhost_access_log,vmware-identity-sts and vmware-identity-sts-perf log files by running these commands:

rm localhost_access_log.*
rm vmware-identity-sts.*
rm vmware-identity-sts-perf.*

Run this command to restart the STS service:

service vmware-stsd restart

Note: Restart the STS service again for new log files to be created in the /storage/log directory.

Note: Ensure to clear \delete the unwanted log bundle collected from VCSA

Additional Information

VMware Skyline Health Diagnostics for vSphere - FAQ
Troubleshooting vCenter Appliance /storage/log directory is 80% or more full
You experience these additional symptoms:

When you run the du -sh command in the /storage/log/vmware/sso directory, you see that several GB of space is utilized.
In the vSphere Web Client, you see the Alerts:
- Available storage for logs /storage/log reached warning threshold - less then 30 % available space
- Available storage for logs/storage/log reached critical threshold - less then 15 % available space
You may also experience issue with /dev/mapper/log_vg-log full

To be alerted when this document is updated, click the Subscribe to Article link in the Actions box

More vCenter Server Appliance disk space issues: vCenter Server Appliance Maintenance Series: Troubleshooting disk space related issues

Read this article in different language here: