sos-access log compression failure causing /storage/log partition to fill on vCenter Server

Products

• VMware vCenter Server

Issue/Introduction

• sps-access logs are not being compressed causing /storage/log to steadily increase in size.
  
  

• The issue can be observed by navigating to the SPS log directory cd /storage/log/vmware/vmware-sps and identifying large, uncompressed access log files using the following command:

  find . -type f -print0 | xargs -0 du -h | sort -rh | head -n 10


• Sample output may show multiple large sps-access-log files consuming substantial disk space:

776M   ./vmware/vmware-sps/sps-access-log.YYYY-MM-DD.log
757M    ./vmware/vmware-sps/sps-access-log.YYYY-MM-DD.log
754M    ./vmware/vmware-sps/sps-access-log.YYYY-MM-DD.log
753M    ./vmware/vmware-sps/sps-access-log.YYYY-MM-DD.log
753M    ./vmware/vmware-sps/sps-access-log.YYYY-MM-DD.log
752M    ./vmware/vmware-sps/sps-access-log.YYYY-MM-DD.log
750M    ./vmware/vmware-sps/sps-access-log.YYYY-MM-DD.log
493M    ./vmware/vmware-sps/sps-access-log.YYYY-MM-DD.log

Environment

vCenter Server 7.x
vCenter Server 8.x

Cause

Logs for the Storage Profile Service, such as sps-access-log and sps-external, produced by services such as the Apache Tomcat Java Servlet for example, might not properly rotate, or compress after rotation, or have no limit on the log file. As a result, unhandled logs fill up the /storage/log directory.

Resolution

The following options are available to remediate or mitigate the issue:
Important: Ensure a snapshot and/or backup of the VCSA or the configuration file is taken before making any changes.

Option 1: Upgrade vCenter Server 

Upgrade to a fixed version of vCenter Server where this issue has been resolved:

• vCenter Server 7.0 U3q or later
• vCenter Server 8.0 U3 or later

Option 2: Disable SPS Access Logging:

1. SSH into the vCenter Server as root.
   
   
2. Backup config file by runningg thr following command:
   cp /usr/lib/vmware-vpx/sps/conf/sms.properties /usr/lib/vmware-vpx/sps/conf/sms.properties_bkp
   
   
3. Edit the SPS configuration file:
   vi /usr/lib/vmware-vpx/sps/conf/sms.properties
   
   
4. Locate the following parameter:
   sms.enabledStr=true
   
   
5. Modify it to disable SPS access logging:
   sms.enabledStr=false


6. Save the file (Esc :wq!)

7. Restart the SPS service to apply the changes:
   service-control --restart vmware-sps


Option 3: Reduce Retention of SPS Access Logs:

1. SSH into the VCSA as root.
   
   
2. Backup config file by runningg thr following command:
   cp /usr/lib/vmware-vpx/sps/conf/sms.properties /usr/lib/vmware-vpx/sps/conf/sms.properties_bkp


3. Edit the configuration file:
   vi /usr/lib/vmware-vpx/sps/conf/sms.properties


4. Locate the retention parameter:
   sms.maxDays=30
   
   
5. Reduce the retention period as required (example below):
   sms.maxDays=14
   
   
6. Save the file (Esc :wq!).
   
   
7. Restart the SPS service:
   service-control --restart vmware-sps

Note: This won't remove existing files over the maxDays so they will need to be removed manually. A cron job can be created to remove files over 30 days.

Additional Information

Patching/Updating VMware vCenter Server through Appliance Management Interface VAMI

VMware vCenter Server 7.0 Update 3q Release Notes