Terminated Pods are Deleted Immediately in VKS Workload Clusters

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

In vSphere Kubernetes Service (VKS) workload clusters, the following symptoms occur:

• Pods in "Completed" or "Failed" status are automatically deleted within a few minutes.
• Rapid deletion makes it difficult to verify logs or collect diagnostic information for troubleshooting.
• Checking the kube-controller-manager arguments reveals that --terminated-pod-gc-threshold is set to 10.

Environment

• vSphere Supervisor 8.0
• vSphere Supervisor 9.0
• vSphere Kubernetes Service (VKS) 3.4 and earlier

Cause

In VKS 3.4 and earlier (ClusterClass builtin-generic-v3.4.0 and older), the terminated-pod-gc-threshold is hardcoded to 10 to protect control plane resources.
This value cannot be modified through the workload cluster YAML manifest.

Resolution

Upgrading to vCenter Server 8.0U3g or later and updating the VKS version will extend the threshold.

1. Upgrade to VKS 3.5 and later: 
   The threshold is increased to 6000, extending Pod retention. Customization is not supported in version 3.5.
   
   
2. Customization in VKS 3.6 and later:
   The kubeControllerManagerConfiguration variable is introduced, allowing customization from the default (6000).
   Specify this in the cluster YAML (topology > variables):

   topology:
     variables:
     - name: kubernetes
       value:
         kubeControllerManagerConfiguration:
           terminatedPodGCThreshold: 12500
   

Note: Increasing this value significantly may impact control plane memory; careful validation is recommended.

Additional Information

• ClusterClass Variable Reference - kubeControllerManagerConfiguration
• VKS のワークロードクラスタで、終了したPodがすぐに削除される