vCenter has a large number of localhost_access log files generated under /storage/log/vmware/eam/web/

Products

• VMware vCenter Server

Issue/Introduction

This is an informational article regarding a known issue discovered with the ESX Agent Manager (EAM) service and log file rotation.

Symptoms:

In vCenter Server 7.0 Update 1 (or newer), a large number of files may be present under the following directory:

• Path: /storage/log/vmware/eam/web/
• Files: localhost_access.*.log

 

Environment

VMware vCenter Server 7.0.x

Cause

This is caused by EAM not having a log file rotation parameter by default, and will leave older files present with no cleanup invoked.

Due to the new functionality of vSphere Clustering Services in 7.0 Update 1, the amount of polling that occurs with EAM causing these log files to generate quickly and can produce a significant quantity of log files.

Resolution

This issue is resolved in vCenter Server 8.0 GA (build number 20519528) and above.

In VMware vCenter Server 7.0 Update 3 (build number 18700403) the localhost_access.*.log files are only zipped and not deleted.

Workaround:

In vCenter Server 7.0 Update 1 and Update 2

On the vCenter Server shell (or SSH session)

1. Backup the file /usr/lib/vmware-eam/web/conf/server.xml
2. Stop the ESX Agent Manager service:

   # service-control --stop vmware-eam

3. Edit /usr/lib/vmware-eam/web/conf/server.xml
4. Locate the line <Valve className="org.apache.catalina.valves.AccessLogValve"
5. Modify the numberOfDays : maxDays="<numberOfDays>" or add  maxDays="<numberOfDays>" if the parameter is not present within the brackets (see example screenshot below)
6. Save the change changes, and restart the ESX Agent Manager service:

   # service-control --start vmware-eam

Example screenshot:

In vCenter Server 7.0 Update 3 and later 7.0 release

vCenter Server 7.0 Update 3 has a partial improvement. The localhost_access.*.log files will be compressed as zip, but not removed.
Remove zip files manually if /storage/log is exhausted.

In vCenter Server 8.x

We will still see a number of localhost_access*.zip files and they will be deleted when a combined size of 128MB is reached.

 

Additional Information

For more information on the vSphere Cluster Service, please refer to the main KB:
vSphere Cluster Services (vCLS) in vSphere 7.0 Update 1 and newer versions

Impact/Risks:

The impact/risk may be that the /storage/log directory fills up, preventing one or more services from functioning. This can lead to instability of vCenter Server.