Setting Process ID Limits for TKGI Clusters

Products

• VMware Tanzu Kubernetes Grid Integrated (TKGi)

Issue/Introduction

For some environments, it is desirable to set a process ID limit for pods in a kubernetes cluster. Normally, this is done through setting the "podPidsLimit" parameter in the kubelet config file. However, for TKGI clusters these settings will not persist through a redeploy or upgrade operation.

Cause

TKGI currently does not have a method for customizing the kubelet config file.

Resolution

In order to update the kubelet configuration in a way that will persist through upgrades, we use Bosh's os-conf release "runtime config" to apply node level file modifications in a pre-start script. The node level change is appending the desired "podPidsLimit" parameter in the kubelet config file and then a restart of the kubelet.

1. Upload os-conf-release:
   
   

   bosh upload-release --sha1 daf34e35f1ac678ba05db3496c4226064b99b3e4 "https://bosh.io/d/github.com/cloudfoundry/os-conf-release?v=22.2.1"

2. Confirm upload:
   
   

   bosh releases | grep os-conf
   os-conf                       22.2.1                         a2154d6

3. Create runtime config: 
   
   NOTE: Please update the exclude.deployments: [] section to exclude any deployments this workaround should NOT be applied to. This section is comma delineated. The below command will create a file called runtime.yaml with the required configuration variables
   
   

   cat <<'EOFA' > runtime.yml
   releases:
   - name: "os-conf"
     version: "22.2.1"
   addons:
   - name: pid-configuration
     exclude:
       deployments: [<excluded deployments comma separated>] #Set which deployments you want to exclude here (non k8's deployments)
     include:
       deployments: [<included deployments comma separated>] #Set which deployments you want to include here
       instance_groups: [worker]
     jobs:
     - name: pre-start-script
       release: os-conf
       properties:
         script: |- #Desired PID limit is set here
           #!/bin/bash
           echo "podPidsLimit: <desired limit>" >> /var/vcap/jobs/kubelet/config/kubeletconfig.yml
           monit restart kubelet
           echo "done"
   EOFA

4. Update bosh runtime config:
   
   

   bosh update-runtime-config runtime.yml

5. Verify 
   
   

   bosh runtime-config

6. Upgrade Cluster:
   
   NOTE: The runtime configuration will only apply to clusters with an upgrade action
   
   

   tkgi upgrade-cluster <CLUSTER_NAME>