net.netfilter.nf_conntrack_max get set back to default since 1.29

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

After upgrading tkgi to 1.20.0 (k8s 1.29.6) modified net.netfilter.nf_conntrack_max with sysctl addon is always set back to default after updating cluster.

 

Before that, the sysctl addon could have been used to set the sysctl 'net.netfilter.nf_conntrack_max' to a higher value on all the nodes used to work.

 

            addons:  
            - include:  
                instance_groups:  
                - master  
                - worker  
              jobs:  
              - name: sysctl  
                properties:  
                  sysctl:  
                  - vm.max_map_count=262145  
                  - net.netfilter.nf_conntrack_max=524288  
                release: os-conf  
              name: pe-sysctl-vm-max-mapcount  
            releases:  
            - name: os-conf  
              version: 22.1.2  

 

Setting the parameter using sysctl and just restart kube-proxy (monit restart kube-proxy) doesn't work either because the parameter is set back to the default value.

Environment

 tkgi 1.20.0 (k8s 1.29.6).

Cause

A code change in k8s 1.29.6. where kube-proxy now applies the default value for the following parameter :

 

--conntrack-max-per-core int32     Default: 32768

 

Resolution

The supported method of modifying the OS is using Bosh Addons https://bosh.io/docs/addons-common/ and https://github.com/cloudfoundry/os-conf-release. There is an existing Addon (https://github.com/cloudfoundry/os-conf-release/tree/master/jobs/pre-start-script) that can be used for modifyingnet.netfilter.nf_conntrack_max. 

Below is an example for setting it up.

Step 1. Upload the latest version 23.0.0 of os-conf release via bosh command:

$ bosh upload-release --sha1 d20772d8ce6e781ceb13cac7df5950bfa4330ba1 "https://bosh.io/d/github.com/cloudfoundry/os-conf-release?v=23.0.0"

Note: If you have the restricted network, you may download the latest version of os-conf release via  https://bosh.io/releases/github.com/cloudfoundry/os-conf-release?all=1 and upload the release via bosh command "bosh upload-release os-conf-release-23.0.0.tgz".

Step 2. Create a new runtime config tkgi-runtime.yml file modifying the desired "maxPerCore" value:

releases:
- name: "os-conf"
  version: "23.0.0"
addons: 
- name: pe-custom-pre-start-proxy-defaut
  include:
    deployments: [service-instance__XXXXXXXXXX]
  exclude:
    instance_groups:
    - master
  jobs:
  - name: pre-start-script
    release: os-conf
    properties:
      script: |-
        #!/bin/bash
        cat >/var/vcap/jobs/kube-proxy/config/config.yml <<EOF
        ---
        apiVersion: kubeproxy.config.k8s.io/v1alpha1
        kind: KubeProxyConfiguration
        clientConnection:
          kubeconfig: "/var/vcap/jobs/kube-proxy/config/kubeconfig"
        clusterCIDR: 19.0.0.0/16
        iptables:
          masqueradeAll: false
          masqueradeBit: 14
          minSyncPeriod: 0s
          syncPeriod: 30s
        conntrack:
          maxPerCore: 90000
        mode: iptables
        portRange: ''
        hostnameOverride:
          EOF

Step 3. Update runtime config:

$ bosh update-runtime-config --name=tkgi-sysctl-config ./tkgi-runtime.yml

Step 4. Upgrade related TKGi clusters via tkgi command: 

tkgi upgrade-cluster <tkgi_cluster_name>

 

Notice: The responsibility of creating and validating these addons lies with the customer. Modifying the OS can have a significant or no impact on the behaviour and performance of the VM so its important the customer validate it.