vCenter Server /storage/log filling up due to localhost_access.log and catalina.log in sso and lookupsvc log directories

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

1. /storage/log is filling up

2. You are receiving constant alerts in vCenter Server that /storage/log is above 80%

3. The output of this command "du -hSx /storage/log |sort -rh |head -30" returns the following directories as the ones using more space:

   • /var/log/vmware/sso/tomcat

   • /var/log/vmware/eam/web

   • /var/log/vmware/lookupsvc/tomcat

   • /var/log/vmware/lookupsvc

4. In those directories there are too many localhost_access*.log and catalina*.log files.

5. localhost_access*.log and catalina*.log are not compressed

Environment

VMware vCenter Server 7.0.x

Cause

Log rotation and compression for these files is not working properly, which can cause the /storage/log partition to be filled up.

Resolution

This issue has been resolved in vCenter Server 7.0 Update 3c. Should you experience issues similar to the ones described in this article despite your vCenter Server already having been updated to 7.0 U3c, please contact the Broadcom Support by opening a support request.

Workaround:

Note: Please ensure to create a fresh backup or offline snapshot (in powered off state) of the vCenter Server Appliance before implementing the steps below. In case the affected vCenter Server Appliance is part of an Enhanced Linked Mode (ELM) replication group, the backup or offline snapshots must be created for all of its replication partners as well. When restoring an ELM vCenter, all members of the ELM replication need to be restored too, otherwise there will be inconstencies in the VMDirectory LDAP database.

1. Copy the files in the attached "configs.zip" file in this KB to the respective file listed below in your vCenter Server Appliance:

   configs.zip                 >  Your vCenter Server
   -----------                    -------------------
   SSO_server.xml              >  /usr/lib/vmware-sso/vmware-sts/conf/server.xml
   SSO_logging.properties      >  /usr/lib/vmware-sso/vmware-sts/conf/logging.properties
   LOOKSVC_server.xml          >  /usr/lib/vmware-lookupsvc/conf/server.xml
   LOOKSVC_logging.properties  >  /usr/lib/vmware-lookupsvc/conf/logging.properties

2. Restart the vmware-stsd and lookupsvc services:

   # service-control --stop vmware-stsd && service-control --start vmware-stsd
   # service-control --stop lookupsvc && service-control --start lookupsvc

3. Copy "tomcat_compress.cron" located in the attached file "tomcat_compress_fixed.zip" in this KB to /etc/cron.d on your vCenter Server appliance.

Additional Information

Please be aware that there is also another issue which might result in a similar behavior with /storage/log running full, but due to a large number of localhost_access.*.log files in /storage/log/vmware/eam/web/ as outlined in KB326212.
This second issue has been fixed in vCenter Server 7.0 Update 3 (build 18700403).

Should you be running an earlier version of vCenter Server 7.0, please do therefore also follow the steps in in KB326212