Terminated pods automatically deleted in VKS 3.4 and earlier.

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Pods which are in terminated state are deleted in the guest cluster.
• There is no parameter defined in the cronjob or the job yaml for the deletion of the pods after a certain time.
• The  kube-controller manager logs of the Guest cluster report the deletion of the pods are being done by PodGC:
  1 gc_controller.go:343] "PodGC is force deleting Pod" logger="pod-garbage-collector-controller" pod="<pod-name>"
1 gc_controller.go:204] "Garbage collecting pods" logger="pod-garbage-collector-controller" numPods=1
1 gc_controller.go:343] "PodGC is force deleting Pod" logger="pod-garbage-collector-controller" pod="<pod-name>"
• kube-controller pod yaml shows the parameter is defined to be 10 by default:
      - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --service-cluster-ip-range=<IP-range>
    - --terminated-pod-gc-threshold=10
• VKS version is 3.4 or earlier.

Environment

VKS 3.4

Cause

• In Kubernetes clusters, such as those in vSphere Kubernetes Service (VKS), a parameter called terminated-pod-gc-threshold controls how many terminated Pods (Succeeded or Failed or completed) are allowed to persist before the garbage collector begins removing them.
• The default value for this threshold is often set to 10 to comply with security standards (e.g., CIS Benchmarks) and to protect control plane resources from exhaustion.

Resolution

This parameter is immutable in VKS 3.4 version and earlier.

This issue is resolved in VKS 3.5.0, ClusterClass builtin-generic-v3.5.0.

When customizing parameters in VKS 3.5 or later, specify them in the cluster's YAML using a format like the following:
 topology:
   class: builtin-generic-v3.5.0
   variables:
   - name: kubernetes
     value:
       kubeControllerManagerConfiguration:
         terminatedPodGCThreshold: <value>

Additional Information

Upgrade the VKS Version 

Upgrade VKS from a Private Registry

Register a New Version of the VKS 

Download Broadcom products, patches and software