How to increase the memory in the VM Operator pod when it is crashing due to insufficient resources at scale
Article ID: 319379
Updated On:
Products
VMware vSphere ESXi
VMware vSphere with Tanzu
Issue/Introduction
This is meant to be shared with customers running a large number of VM workloads who are experiencing issues realizing those workloads with VM Operator due to the pod crashing due to insufficient memory allocated to the pod.
Symptoms:
If VirtualMachine resources are taking a long time to be realized when at scale (ex. thousands of VMs), then it could be due to the VM Operator pod crashing due to insufficient memory. To determine if this is the case, please follow these steps:
1. SSH into the vCenter appliance:
2. Follow KB 90194 to ssh into the Supervisor Control Plane VM as root.
3. Check to see if VM Operator has crashed due to an Out of Memory (OOM) error:
If OOMKilled is in the output from the above command, then the pod was terminated due to lack of sufficient memory.
Symptoms:
If VirtualMachine resources are taking a long time to be realized when at scale (ex. thousands of VMs), then it could be due to the VM Operator pod crashing due to insufficient memory. To determine if this is the case, please follow these steps:
1. SSH into the vCenter appliance:
ssh root@<VCSA_IP>
2. Follow KB 90194 to ssh into the Supervisor Control Plane VM as root.
PLEASE NOTE: When on the supervisor control plane VM you have permissions to permanently damage the cluster. If VMware Support finds evidence of a customer making changes to the supervisor cluster from the SV VM, they may mark your cluster as unsupported and require you redeploy the entire vSphere with Tanzu solution. Only use this session to test networks, look at logs, and run kubectl logs/get/describe commands. Do not deploy, delete, or edit anything from this session without the express permissions from VMware Support or specific instructions about what exactly you need to deploy/delete/edit from a kb.
3. Check to see if VM Operator has crashed due to an Out of Memory (OOM) error:
kubectl -n vmware-system-vmop \
describe pods -l control-plane=controller-manager | \
grep -F OOMKilled
describe pods -l control-plane=controller-manager | \
grep -F OOMKilled
If OOMKilled is in the output from the above command, then the pod was terminated due to lack of sufficient memory.
Environment
VMware vSphere 7.0 with Tanzu
Cause
The VM Operator pod is allocated 400Mi (MB) in vSphere with Tanzu prior to 8.0 U2c, and was increased to 500Mi starting at 8.0 U2c. This will further be increased to 550Mi in 8.0 U3 to accommodate growing consumption of VMOP resources.
The recent addition of Prometheus metrics to the VMOP pod increased the amount of memory consumed during active reconciliation. This "burst" requirement exceeds the hard limit of 400Mi.
The recent addition of Prometheus metrics to the VMOP pod increased the amount of memory consumed during active reconciliation. This "burst" requirement exceeds the hard limit of 400Mi.
Resolution
In vSphere 8.0 U2c the VM Operator pod's memory limit is increased to 500Mi.
It is possible user workloads will still go over the 500Mi limit. If OOM errors are still encountered on later revisions of vSphere with Tanzu, it maybe be required to increase the limit greater than 500Mi, this will be environmentally dependent. Follow the workaround below to increase the limit accordingly.
Workaround:
The memory limit of the VM Operator pod can be increased. The following steps describe how to increase the limit to 500Mi (MB) or higher as needed:
1. SSH into the vCenter appliance:
2. Follow KB 90194 to ssh into the Supervisor Control Plane VM as root.
3. Scale down the VM Operator deployment so no pods are running:
4. Increase the memory limit to 500MB (or greater if required) for the VM Operator pod using the following command:
5. Scale the replicas back to their original value:
Everything should now work correctly. If OOM errors are still encountered, increase the memory limit in increments of 50Mi until the VMOP pods run normally.
It is possible user workloads will still go over the 500Mi limit. If OOM errors are still encountered on later revisions of vSphere with Tanzu, it maybe be required to increase the limit greater than 500Mi, this will be environmentally dependent. Follow the workaround below to increase the limit accordingly.
Workaround:
The memory limit of the VM Operator pod can be increased. The following steps describe how to increase the limit to 500Mi (MB) or higher as needed:
1. SSH into the vCenter appliance:
ssh root@<VCSA_IP>
2. Follow KB 90194 to ssh into the Supervisor Control Plane VM as root.
PLEASE NOTE: When on the supervisor control plane VM you have permissions to permanently damage the cluster. If VMware Support finds evidence of a customer making changes to the supervisor cluster from the SV VM, they may mark your cluster as unsupported and require you redeploy the entire vSphere with Tanzu solution. Only use this session to test networks, look at logs, and run kubectl logs/get/describe commands. Do not deploy, delete, or edit anything from this session without the express permissions from VMware Support or specific instructions about what exactly you need to deploy/delete/edit from a kb.
3. Scale down the VM Operator deployment so no pods are running:
kubectl -n vmware-system-vmop \
scale deployment vmware-system-vmop-controller-manager \
--replicas 0
scale deployment vmware-system-vmop-controller-manager \
--replicas 0
4. Increase the memory limit to 500MB (or greater if required) for the VM Operator pod using the following command:
kubectl -n vmware-system-vmop \
patch deployment vmware-system-vmop-controller-manager \
-p '{"spec":{"template":{"spec":{"containers":[{"name":"manager","resources":{"limits":{"memory":"500Mi"}}}]}}}}'
patch deployment vmware-system-vmop-controller-manager \
-p '{"spec":{"template":{"spec":{"containers":[{"name":"manager","resources":{"limits":{"memory":"500Mi"}}}]}}}}'
5. Scale the replicas back to their original value:
kubectl -n vmware-system-vmop \
scale deployment vmware-system-vmop-controller-manager \
--replicas 3
scale deployment vmware-system-vmop-controller-manager \
--replicas 3
Everything should now work correctly. If OOM errors are still encountered, increase the memory limit in increments of 50Mi until the VMOP pods run normally.
Additional Information
Reported in 7.0 U3e
Feedback
Yes
No
Powered by
