Log Disk Exhaustion caused by wcp stdstream.log

Products

• VMware vCenter Server

Issue/Introduction

To prevent disk space exhaustion in the /storage/log partition of vCenter Server

Symptoms:

• vSphere Client is displaying a warning "Log Disk Exhaustion on <vcenter-name>" on the summary page for the vCenter Server

• Upon investigation the folder /var/log/vmware/wcp is found to be either one of or the largest folder on the /storage/log partition

• Looking in the folder, there is one or more very large stdstream.log files, similar to the example below:

  # ls -althrS /var/log/vmware/wcp | tail -5
  1.4M  -rw-------  1 wcp  lwisRegReader 1.4M <DATE> <TIME> stdstream.log-1.stderr
  34M  -rw-------  1 wcp  lwisRegReader  34M <DATE> <TIME>stdstream.log-4.stderr
  111M  -rw-------  1 wcp  lwisRegReader 111M <DATE> <TIME> stdstream.log.stderr
  1.1G  -rw-------  1 wcp  lwisRegReader 1.1G <DATE> <TIME> stdstream.log-3.stderr
  1.6G  -rw-------  1 wcp  lwisRegReader 1.6G <DATE> <TIME> stdstream.log-0.stderr

   

Environment

VMware vCenter Server 7.0.x

Cause

The stdstream.log.stderr logs are currently not rotated by size, but only when the wcp service restarts. When wcp runs for longer amounts of this, their size can therefore grow quite large, thus allocating large portions of the space available in /storage/log, and potentially resulting in the space getting exhausted.

Resolution

Currently there is no permanent solution available.
VMware engineering is aware of this issue and working towards a fix in a future release

Workaround:

To mitigate this problem

1. Please take a fresh backup or powered off snapshot of the vCenter Server Appliance

2. Remove all stdstream.log* files from the folder

# rm /storage/log/vmware/wcp/stdstream.log*.stderr 

3. Restart the wcp service:

# service-control --stop wcp && service-control --start wcp

Additional Information

See also: 

Storage logs partition full due to large stream.log-x.stderr files