Helm-controller and source-controller pods remain in a Pending state on the Control Plane node in VKS 3.7 and VKr 1.33.x
Article ID: 449339
Updated On:
Products
Issue/Introduction
After a new cluster is created or an existing cluster is upgraded to vSphere Kubernetes Service (VKS) 3.7 on a single Control Plane node topology, the helm-controller and source-controller pods within the vmware-system-helm namespace remain in a Pending state.
Execution of the kubectl describe pod <pod-name> command on the pending helm-controller pod reveals the following error: Insufficient cpu scheduling
Environment
vSphere Kubernetes Service 3.7.0
vSphere Kubernetes Release 1.33.x
Single Control Plane node topology (configured with 2 or fewer CPUs, e.g., best-effort-small or smaller VM classes)
Cause
Starting in VKS 3.7, the platform deploys the helm-controller and source-controller by default to support Helm-based add-ons. To support multi-OS clusters, these controller pods are hard-coded to schedule strictly onto Control Plane nodes.
In older releases such as VKr 1.33.x, the core system add-ons possess higher default CPU resource allocations compared to newer VKr versions. When these core add-ons are combined with the new helm and source controllers on a single Control Plane node with 2 or fewer CPUs, the node exhausts its allocatable CPU space, which prevents the helm-controller and source-controller pods from scheduling.
Resolution
To resolve this issue, apply one of the following three options:
Option 1: Use a newer vSphere Kubernetes Release (Recommended)
Deploy or upgrade the cluster using VKr version 1.34.1 or higher. These versions utilize optimized, reduced CPU allocations for core system add-ons, leaving enough resource headroom on a single Control Plane node for the Helm controllers to run successfully.
Option 2: Increase Control Plane Node Resources
If the environment must remain on VKr 1.33.x, increase the resource capacity of the Control Plane by performing one or both of the following actions:
Configure a multi-node Control Plane topology instead of a single node.
Configure a larger VM class for the Control Plane node that provides more than 2 CPUs (such as best-effort-large or higher).
Option 3: Opt-out of the default helm-controller deployment
If workloads do not require the default Helm-based add-ons capability, the platform can be prevented from deploying these controllers by adding the following label to the Cluster Custom Resource (CR) when creating a new cluster or before upgrading an existing cluster:
Feedback
