The vmware-system-vmop-controller-manager pods on a vSphere Supervisor are stuck in a CrashLoopBackOff state due to OOMKilled

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• vmware-system-vmop-controller-manager (VM operator) pods are restarting and remain stuck in a CrashLoopBackOff state.

• Describing an affected vmware-system-vmop-controller-manager pod returns a container is failing due to OOMKilled:
  
  kubectl describe pod -n <namespace> <pod name>
finishedAt: "YYYY-MM-DDTHH:MM:SSZ"
        reason: OOMKilled
        startedAt: "YYYY-MM-DDTHH:MM:SSZ"

Environment

• vSphere 8.x

• vSphere Supervisor

Cause

The affected VM operator pods default memory limits are unable to keep up with the large amount of resources needed by a large vSphere Supervisor environment.

Resolution

Workaround:

Increase the memory limit of the VM Operator pod in 50Mi increments until the pods start and remain in a Running status.

The following steps detail how to increase the limit to 600Mi (MB) or higher as needed:

1. SSH into the vCenter appliance: 

ssh root@<VCSA_IP>

2. Follow KB article 90194  to connect into the Supervisor Control Plane VM as root via SSH.
 

PLEASE NOTE: When on the supervisor control plane VM you have permissions to permanently damage the cluster. If VMware Support finds evidence of a customer making changes to the supervisor cluster from the SV VM, they may mark your cluster as unsupported and require you redeploy the entire vSphere with Tanzu solution.  Only use this session to test networks, look at logs, and run kubectl logs/get/describe commands. Do not deploy, delete, or edit anything from this session without the express permissions from VMware Support or specific instructions about what exactly you need to deploy/delete/edit from a kb.

 

 

3. Describe the vmware-system-vmop-controller-manager deployment and determine the existing memory configuration:

Containers:
   manager:
    Image:       localhost:5000/vmware/vmop:0.1.0-bdffdb8-1.8.6-########-########
    Ports:       9878/TCP, 9445/TCP
    Host Ports:  0/TCP, 0/TCP
    Command:
      /manager
    Args:
      --metrics-addr=127.0.0.1:8083
    Limits:
      cpu:     750m
      memory:  550Mi

 

 

4. Scale down the VM Operator deployment so no pods are running:
 

kubectl -n vmware-system-vmop scale deployment vmware-system-vmop-controller-manager --replicas 0

 

5. Increase the memory limit by 50Mi for the VM Operator pod using the following command (this example is increasing the original 550Mi configuration to 600Mi):
 

kubectl -n vmware-system-vmop patch deployment vmware-system-vmop-controller-manager -p '{"spec":{"template":{"spec":{"containers":[{"name":"manager","resources":{"limits":{"memory":"600Mi"}}}]}}}}'

 

6. Scale the replicas back to their original value:
 

kubectl -n vmware-system-vmop scale deployment vmware-system-vmop-controller-manager --replicas 2

 

If OOMKilled errors are still encountered, increase the memory limit in increments of 50Mi until the VMOP pods run normally.

Additional Information

Verify if the TKGS content library has been incorrectly attached on the vSphere namespace level .
This issue can also occur due to large number of VirtualMachineImages in the vSphere Namespaces .

To check the number of VirtualMachineImages :
kubectl get virtualmachineimages -A --no-headers | wc -l

 

Refer : Supervisor Control Plane VMs experience high CPU and Memory with vmop-controller-manager OOMKilled