Kubernetes Networking Add-on PriorityClass Eviction Issues in TKGm

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform
• VMware Tanzu Kubernetes Grid Management

Issue/Introduction

Networking add-ons, specifically Multus and Whereabouts, are deployed with a default priority of 0 in TKGm environments. Resource pressure on nodes can cause these critical system components to be preempted by other workloads, leading to instability or service failure.

Environment

TCA: 3.4, TCP:5.1

Cause

Networking add-ons (Multus, Whereabouts) are configured with default Priority 0. Assigning higher PriorityClass values to custom workloads triggers the Kubernetes scheduler to preempt these lower-priority system pods during resource contention.

Resolution

To prioritize large, resource-intensive workloads without risking the eviction of Multus or Whereabouts, implement a custom PriorityClass using the preemptionPolicy: Never setting. This configuration ensures workloads are placed at the front of the scheduling queue without actively terminating running pods that have lower priority.

Apply the following PriorityClass configuration to your cluster:

yaml

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority-non-preempting
value: 1000000
preemptionPolicy: Never
globalDefault: false
description: "Places pods at the front of the queue without preempting running pods."

Benefits

• Prioritized Scheduling: Workloads are queued ahead of lower-priority pods.
• Platform Stability: The preemptionPolicy: Never setting prevents these workloads from preempting critical infrastructure pods (Priority=0).

 

Additional Information

For future architectural planning, migrating to VCF 9.x vSphere Kubernetes Service (VKS) provides native PriorityClass alignments by default. Broadcom currently has no plans to modify the default PriorityClass for these specific add-ons in TKGm.