Log Disk Exhaustion caused by wcp stdstream.log

Products

• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

• To prevent disk space exhaustion in the /storage/log partition of vCenter Server

• vSphere Client is displaying a warning "Log Disk Exhaustion" on the summary page for the vCenter Server

• Upon investigation the folder /var/log/vmware/wcp is found to be either one of or the largest folder on the /storage/log partition

• Looking in the folder, there is one or more very large stdstream.log files, similar to the example below:

  # ls -althrS /var/log/vmware/wcp | tail -5
  1.4M  -rw-------  1 wcp  lwisRegReader 1.4M <DATE> <TIME> stdstream.log-1.stderr
  34M  -rw-------  1 wcp  lwisRegReader  34M <DATE> <TIME>stdstream.log-4.stderr
  111M  -rw-------  1 wcp  lwisRegReader 111M <DATE> <TIME> stdstream.log.stderr
  1.1G  -rw-------  1 wcp  lwisRegReader 1.1G <DATE> <TIME> stdstream.log-3.stderr
  1.6G  -rw-------  1 wcp  lwisRegReader 1.6G <DATE> <TIME> stdstream.log-0.stderr

Environment

• VMware vCenter Server 7.0.x
• VMware vCenter Server 8.0.x

Cause

The Workload Control Plane (WCP) service generates stdstream.log.stderr logs which do not rotate based on size. These logs rotate only when the service restarts. Over extended uptime, these files can grow indefinitely, leading to partition exhaustion.

Resolution

This issue is resolved in vCenter Server 7.0 U3v and vCenter Server 8.0 U3.

To workaround the disk exhaustion and mitigate future growth, follow these steps:

1. Log in to the vCenter Server Appliance via SSH or the Appliance Shell as root
2. Identify the large log files in the WCP directory (focus on files with the largest byte/gigabyte count) : ls -althrS /storage/log/vmware/wcp | tail -10
3. Clear the content of the identified large .stderr files: echo "" > /storage/log/vmware/wcp/stdstream.log-0.stderr
   Note: Using echo "" > is preferred over deleting the file. It empties the file content while keeping the existing file handle active. This allows the system to reuse the file without requiring an immediate service restart, preventing potential issues with broken file handles or service interruptions. Repeat this command for other large stdstream.log-x.stderr files identified in Step 2.
4. Restart the WCP service to initialize new log files: service-control --stop wcp && service-control --start wcp
5. Verify that space has been reclaimed: df -h /storage/log

Additional Information

Storage logs partition full due to large stream.log-x.stderr files