Customizing the kubelet parameters for log retention in TKG

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Pods generating huge flux of logs and to avoid missing any logs due to log rotation, by updating the default values for kubelet parameters containerLogMaxSize and containerLogMaxFiles

Environment

VMware Tanzu Kubernetes Grid Plus 1.x
VMware Tanzu Kubernetes Grid 1.x

Cause

By default kubelet will rotate the logs after a maximum of 5 log files each of which sized to 10Mi and once it reaches this limit, the logs will be rotated then we can see only the contents of the recent logs.

Resolution

Customizing these values onto existing clusters, you need to edit the kubeadmconfigtemplates for the worker machine deployment and kubeadmcontrolplanes then you will need to recreate the nodes for the changes to be reflected.

Procedure for existing clusters:
Target to the management cluster context and edit the kubeadmcontrolplane for the workload cluster where you want to change the default values for container-log-max-size and container-log-max-files with your custom values as shown in the below configuration. After editing the KCP by adding your custom values, it will roll out the control plane nodes for the cluster.

    initConfiguration:
      ...
      nodeRegistration:
        kubeletExtraArgs:
          ...
          container-log-max-files: "8" 
          container-log-max-size: "20Mi"
       
    joinConfiguration:
      ...
      nodeRegistration:
        ...
        kubeletExtraArgs:
          ...
          container-log-max-files: "8"
          container-log-max-size: "20Mi"

Once that roll out is completed then edit the kubeadmconfigtemplate for the cluster and add the same parameters with your custom values as shown below: 

      joinConfiguration:
        nodeRegistration:
          ...
          kubeletExtraArgs:
            ...
            container-log-max-files: "8"
            container-log-max-size: "20Mi"

To make these changes pushed to the worker nodes in the cluster use the below command by replacing with the machinedeployment name to make the nodes of the cluster to be patched with the changes:

kubectl patch machinedeployment <machine deployment>  --type merge -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}"

This will roll out the nodes one at a time and we can check whether the changes has taken effect or not by ssh to the node and grep for the kubelet process as shown below: 

ps -ef | grep kubelet
root         848       1  3 Nov29 ?        00:42:17 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cloud-provider=external --container-log-max-files=8 --container-log-max-size=20Mi

Note: If there are multiple workload cluster, please follow the same procedure editing the kcp and the kubeadmconfigtemplates for each cluster and patch the nodes.

Procedure for Newly creating Clusters:

Add the below mentioned overlay by updating with your custom values to the ".tanzu/tkg/providers/infrastructure-IAAS/ytt/ "  folder and trigger the cluster creation then the newly created clusters will pick the overlay and the values with the propagated to the nodes. Where IAAS is vsphere, aws, or azure.

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#@overlay/match by=overlay.subset({"kind":"KubeadmConfigTemplate"})
---
spec:
  template:
    #@overlay/match missing_ok=True
    spec:
      joinConfiguration:
        nodeRegistration:
          kubeletExtraArgs:
            #@overlay/match missing_ok=True
            container-log-max-files: "8"
            #@overlay/match missing_ok=True
            container-log-max-size: "20Mi"

#@overlay/match by=overlay.subset({"kind":"KubeadmControlPlane"})
---
spec:
  kubeadmConfigSpec:
    #@overlay/match missing_ok=True
     initConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          #@overlay/match missing_ok=True
          container-log-max-files: "8"
          #@overlay/match missing_ok=True
          container-log-max-size: "20Mi"
     joinConfiguration:
      nodeRegistration:
        kubeletExtraArgs:
          #@overlay/match missing_ok=True
          container-log-max-files: "8"
          #@overlay/match missing_ok=True
          container-log-max-size: "20Mi"

Note: In my example I have used container-log-max-files as 8 and container-log-max-size as 20Mi.