vCenter Appliance is running low on memory due to wcp service

Products

VMware vCenter Server Tanzu Kubernetes Runtime VMware vSphere Kubernetes Service

Issue/Introduction

The Workload Control Plane (wcp) service is experiencing excessive memory consumption causing vCenter to report Memory Exhaustion.

This high utilization creates significant resource contention and risks overall vCenter instability.

The service appears to be stuck in an inefficient processing loop, characterized by high-frequency logging and repetitive, complex database interactions.

The following symptoms are observed:

The affected vCenter has a Memory Exhaustion Alarm similar to one of the following, where <vCenter> is the name of the affected vCenter:

Memory Exhaustion on <vCenter>
Alarm 'Memory Exhaustion on <vCenter>' on Datacenters changed from Green to Yellow
Alarm 'Memory Exhaustion on <vCenter>' on Datacenters changed from Yellow to Red

When viewing the Health Status of the affected vCenter from VMware Appliance Management Interface (VAMI), Memory has an Alert showing:
```
Appliance is running low on memory.
Add more memory to the machine.
```
When SSH into the vCenter VM that is experiencing memory issues, the following command returns that a process called "wcp" is using a large amount of memory.
This command outputs the top 10 processes using the most memory on the vCenter VM in descending order.
```
ps -aux --sort=-%mem | head

USER         PID   %CPU  %MEM
wcp          <PID> <CPU> <MEM>
```
While SSH into the affected vCenter VM, and checking the "top" command, we see wcp consuming most of the vCenter Server's memory consumption:
```
top
```

Environment

vCenter 8.X

Cause

The root cause is a memory leak and inefficient data processing loop within the Workload Control Plane service's internal cache management.

In affected versions of vCenter 8.0, the service repeatedly triggers complex SQL queries against the kube_config_maps table to parse metadata across multiple cluster domains, failing to effectively clear or manage the resulting data in memory.

Resolution

Permanent Fix

Upgrade vCenter Server to version 8.0 Update 3g (8.0P06) or later which contains the specific fix for wcpsvc memory management and database query optimization.

Workaround (Immediate Mitigation)

If an upgrade cannot be performed immediately, restart the WCP service to clear the leaked memory and reset the processing loop:

Log in to the vCenter Server Appliance (VCSA) via SSH as root.
Execute the following command to restart the WCP service:
Restarting the wcp service does not affect functionality other than a brief alert appearing on the vCenter web UI.
```
service-control --restart wcp
```
Confirm that the process is running afterwards:
```
service-control --status wcp
```

Check that wcp process memory usage has reduced:

ps -aux --sort=-%mem | grep "vmware-wcp/wcpsvc"

wcp <PID> <CPU> <MEM>

Acknowledge or Reset the Memory Exhaustion alarms accordingly.