How to increase the memory in the VM Operator pod when it is crashing due to insufficient resources at scale

Products

VMware vSphere ESXi VMware vSphere Kubernetes Service

Issue/Introduction

This is meant to be shared with customers running a large number of VM workloads who are experiencing issues realizing those workloads with VM Operator due to the pod crashing due to insufficient memory allocated to the pod.

Symptoms:
If VirtualMachine resources are taking a long time to be realized when at scale (ex. thousands of VMs), then it could be due to the VM Operator pod crashing due to insufficient memory. To determine if this is the case, please follow these steps:

1. SSH into the vCenter appliance:

ssh root@<VCSA_IP>

2. Follow KB 90194 to ssh into the Supervisor Control Plane VM as root.

PLEASE NOTE: When on the supervisor control plane VM you have permissions to permanently damage the cluster. If VMware Support finds evidence of a customer making changes to the supervisor cluster from the SV VM, they may mark your cluster as unsupported and require you redeploy the entire vSphere with Tanzu solution. Only use this session to test networks, look at logs, and run kubectl logs/get/describe commands. Do not deploy, delete, or edit anything from this session without the express permissions from VMware Support or specific instructions about what exactly you need to deploy/delete/edit from a kb.

3. Check to see if VM Operator has crashed due to an Out of Memory (OOM) error:

kubectl -n vmware-system-vmop \
describe pods -l control-plane=controller-manager | \
grep -F OOMKilled

If OOMKilled is in the output from the above command, then the pod was terminated due to lack of sufficient memory.

Environment

VMware vSphere 7.0 with Tanzu

Cause

The VM Operator pod is allocated 400Mi (MB) in vSphere with Tanzu prior to 8.0 U2c, and was increased to 500Mi starting at 8.0 U2c. This will further be increased to 550Mi in 8.0 U3 to accommodate growing consumption of VMOP resources.

The recent addition of Prometheus metrics to the VMOP pod increased the amount of memory consumed during active reconciliation. This "burst" requirement exceeds the hard limit of 400Mi.

Resolution

In vSphere 8.0 U2c the VM Operator pod's memory limit is increased to 500Mi.

It is possible user workloads will still go over the 500Mi limit. If OOM errors are still encountered on later revisions of vSphere with Tanzu, it maybe be required to increase the limit greater than 500Mi, this will be environmentally dependent. Follow the workaround below to increase the limit accordingly.

Workaround:
The memory limit of the VM Operator pod can be increased. The following steps describe how to increase the limit to 500Mi (MB) or higher as needed:

1. SSH into the vCenter appliance:

ssh root@<VCSA_IP>

2. Follow 90194 to ssh into the Supervisor Control Plane VM as root.

PLEASE NOTE: When on the supervisor control plane VM you have permissions to permanently damage the cluster. If VMware Support finds evidence of a customer making changes to the supervisor cluster from the SV VM, they may mark your cluster as unsupported and require you redeploy the entire vSphere with Tanzu solution. Only use this session to test networks, look at logs, and run kubectl logs/get/describe commands. Do not deploy, delete, or edit anything from this session without the express permissions from VMware Support or specific instructions about what exactly you need to deploy/delete/edit from a kb.

3. Scale down the VM Operator deployment so no pods are running:

kubectl -n vmware-system-vmop \
scale deployment vmware-system-vmop-controller-manager \
--replicas 0

4. Increase the memory limit to 500MB (or greater if required) for the VM Operator pod using the following command:

kubectl -n vmware-system-vmop \
patch deployment vmware-system-vmop-controller-manager \
-p '{"spec":{"template":{"spec":{"containers":[{"name":"manager","resources":{"limits":{"memory":"500Mi"}}}]}}}}'

5. Scale the replicas back to their original value:

kubectl -n vmware-system-vmop \
scale deployment vmware-system-vmop-controller-manager \
--replicas 3

Everything should now work correctly. If OOM errors are still encountered, increase the memory limit in increments of 50Mi until the VMOP pods run normally.

Additional Information

Reported in 7.0 U3e