Supervisor Control Plane VMs experience high CPU and Memory with vmop-controller-manager OOMKilled
search cancel

Supervisor Control Plane VMs experience high CPU and Memory with vmop-controller-manager OOMKilled

book

Article ID: 445521

calendar_today

Updated On:

Products

VMware vSphere Kubernetes Service VMware Tanzu Platform - Kubernetes

Issue/Introduction

VMware vSphere with Tanzu Supervisor Control Plane VMs (CPVMs) regularly spike to full CPU and memory utilization, leading to unresponsiveness until manually restarted.

Key symptoms and error messages include:

  • The etcd database grows unusually large (e.g., 1.4 GB to 1.6 GB) and reaches maximum capacity very quickly.

  • The vmop-controller-manager pod repeatedly fails and enters a CrashLoopBackOff state with an OOMKilled (Exit code 137) error.

  • CRITICAL SYMPTOM: The kube-apiserver consumes exorbitant CPU and memory resources. Running the top command on the CPVMs clearly identifies kube-apiserver as the primary consumer causing the resource exhaustion, heavily impacting node stability and often exceeding 300% CPU:

%Cpu2  :  23.1/73.9   97[|||||||||||||||||||||||||||||||||||||||||         ]   
GiB Mem :  99.5/23.5    [                                                  ]   

  PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND
18812 root      20   0   19.0g  14.8g 377.0  62.9  21:08.02 S kube-apiserver --admission-control-config-file=/etc/vmware/wcp/admission-control.yaml...
22373 root      20   0  7805.8m  1.7g  36.2   7.1   0:29.09 S kubectl delete events --field-selector type=Normal -A

 

Environment

VMware vSphere with Tanzu (WCP)
VKS 8.x, 9.x

Cause

The root cause is an improper association of the TKG/TKR Subscribed Content Library directly to individual vSphere Namespaces instead of exclusively at the Supervisor Cluster level.

When associated per-namespace in an environment with numerous namespaces, it triggers a massive multiplication of VirtualMachineImage custom resources (CRs) within etcd. This bloat can be directly verified by checking the object count in etcd, which will reveal an exceptionally massive number of keys (e.g., over 14,000 combined) under vmoperator.vmware.com and imageregistry.vmware.com:

# etcdctl get /registry --prefix --keys-only | grep -v ^$ | awk -F '/' '{ h[$3]++ } END {for (k in h) print h[k], k}' | sort -nr
14100 vmoperator.vmware.com
# imageregistry.vmware.com
# secrets
# vmware.infrastructure.cluster.x-k8s.io
# cns.vmware.com
# cluster.x-k8s.io
# cert-manager.io
# bootstrap.cluster.x-k8s.io
# run.tanzu.vmware.com
# rolebindings
# imagecontroller.vmware.com
# controlplane.cluster.x-k8s.io

This severe bloat exhausts etcd capacity and causes the vmop-controller-manager to hit its memory limits and crash (OOMKilled - Exit code 137).

The massive volume of duplicate objects and continuous churn forces the kube-apiserver to consume all available CPVM compute resources (as highlighted in the top output) to process the excessive API requests, leading to the full node exhaustion.

Resolution

Workaround

To temporarily stabilize the Control Plane VMs and allow the UI/API to function while the Content Library configuration is corrected, perform the following steps to clean up etcd bloat:

  1. Reboot the older/affected Supervisor Control Plane nodes if they are entirely locked up.

  2. Connect to the Supervisor Control Plane VMs via SSH.

  3. Temporarily stop the kube-apiserver on all CPVM nodes to halt resource churn: mv /etc/kubernetes/manifests/kube-apiserver.yaml /root/

  4. Identify the current etcd database revision: rev=$(etcdctl endpoint status --write-out="json" | grep -oE '"revision":[0-9]+' | cut -d: -f2)

  5. Compact the etcd database using the identified revision: etcdctl --command-timeout=60s compact $rev

  6. Defragment the etcd cluster: etcdctl --command-timeout=60s defrag --cluster

  7. Delete any pods currently in a Terminating or CrashLoopBackOff state.

  8. Verify the database size has successfully decreased (e.g., to ~1.2 GB): etcdctl --cluster=true endpoint status -w table

  9. Restore the kube-apiserver manifest to bring the API back online: mv /root/kube-apiserver.yaml /etc/kubernetes/manifests/

  10. Verify that all core pods stabilize into a Running state: kubectl get pods -A | grep -vE "Running|Complete"

 

To permanently resolve the issue, the per-namespace library subscriptions must be removed.

  1. Navigate to the vCenter Server UI.

  2. Remove the TKG/TKR Subscribed Content Library associations from all individual vSphere Namespaces.

  3. Ensure the TKG/TKR Content Library is associated exclusively at the Supervisor cluster level.

  4. Once the per-namespace associations are removed, the duplicate VirtualMachineImage CRs will be automatically garbage-collected. This shrinks the etcd database to a sustainable size and stops the massive kube-apiserver CPU and memory spikes.

Additional Information

For more details regarding proper Subscribed Content Library configuration for vSphere with Tanzu, reference the following Knowledge Base article: How to add or update VKS Content Libraries (https://knowledge.broadcom.com/external/article/411775/how-to-add-or-update-vks-content-librari.html)

 

Powered by Wolken Software