Supervisor upgrade failed with VmOperator : Component Configuration error

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

While upgrading the supervisor, the VmOperator component fails with below error:

VmOperator configured : Configuration error (since mm/dd/yyyy, hh:mm:ss PM)
Component Configuration error: Component VmOperatorUpgrade failed: Failed to run command: ['kubectl', 'rollout', 'status', 'deployment', 'vmware-system-vmop-controller-manager', '-n', 'vmware-system-vmop', '--timeout=3m', '--watch=true'] ret=1 out=Waiting for deployment "vmware-system-vmop-controller-manager" rollout to finish: 1 of 2 updated replicas are available... Waiting for deployment "vmware-system-vmop-controller-manager" rollout to finish: 0 of 2 updated replicas are available... err=error: timed out waiting for the condition

Environment

• Supervisor 9.x
• Supervisor 8.x

Cause

The error indicates that the vmware-system-vmop-controller-manager deployment (the VM Operator) is failing to roll out within the 3-minute timeout period during the upgrade. This is a common issue when the VM Operator pods encounter resource limits or pending states due to scale or configuration changes in vSphere 9.1.

Resolution

1. Check the Pod status: 

   1. kubectl get pods -n vmware-system-vmop

   2. kubectl describe pod -n vmware-system-vmop <pod-name>
      
      Note : Look for events like OOMKilled (Out of Memory) or ImagePullBackOff.

2. Increase Memory Limits: If you see OOMKilled or high memory usage, increase the memory limits for the controller manager to allow it to reconcile the larger 9.1 inventory:

   1. kubectl -n vmware-system-vmop patch deployment vmware-system-vmop-controller-manager -p '{"spec":{"template":{"spec":{"containers":[{"name":"manager","resources":{"limits":{"memory":"800Mi"}}}]}}}}'
      
      Note : Standard default is often 500Mi; increasing to 800Mi or 1000Mi is often required in high-scale environments
      Note : Reference KB : How to increase the memory in the VM Operator pod when it is crashing due to insufficient resources at scale

3. Scale Down/Up to Clear Cache: Pods stuck in a "Configuring" loop / "crashLoopback" or have cache issues:

   1. Wait for 30 second after executing this : kubectl scale deployment vmware-system-vmop-controller-manager -n vmware-system-vmop --replicas=0

   2. kubectl scale deployment vmware-system-vmop-controller-manager -n vmware-system-vmop --replicas=2