Log Rotation with containerd from TKGi 1.14.x or higher version

Products

• VMware Tanzu Kubernetes Grid Integrated (TKGi)

Issue/Introduction

How to customize container logs configuration, such as max log size or max number of log files through:

1. Kubelet config file (recommended) -> containerLogMaxSize and containerLogMaxFiles
2. Kubernetes profiles (deprecated) -> container-log-max-size and container-log-max-files

Environment

VMware Tanzu Kubernetes Grid Integrated Edition 1.14 and above.

Resolution

• The default kubelet configuration is at "/var/vcap/jobs/kubelet/config/kubeletconfig.yml" inside the nodes.
• Kubelet config in TKGi v1.14 or higher uses API v1beta1. Default setting is mentioned in upstream K8s document below: https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/

There are two ways to set up the max log files' size and max number of log files.

1. Editing the kubelet config file (recommended):

   Inside the node where we want to make the kubelet changes:
   
   1. # vim /var/vcap/jobs/kubelet/config/kubeletconfig.yml
   
   2. Add the kubelet config file parameter you want to edit (list of all parameters), for example, containerLogMaxSize and/or containerLogMaxFiles.
   
   E.g.:
   
   # cat /var/vcap/jobs/kubelet/config/kubeletconfig.yml
---
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 0s
enabled: true
x509:
clientCAFile: "/var/vcap/jobs/kubelet/config/kubelet-client-ca.pem"
authorization:
mode: Webhook
clusterDNS:
- <>
clusterDomain: cluster.local
containerLogMaxSize: 100Mi
   
   3. Restart the kubelet job:
   # monit restart kubelet
   
   Please note the above change is not persistent and will be reverted if the node is recreated.
   
   To make the log config changes persistent across node recreation and apply them to all the nodes in a Bosh deployment at once, follow the steps below:
   
   1. Get the cluster manifest:
   $ bosh -d service-instance_<xxxx> manifest > service-instance_<xxxx>.manifest
   
   2. Create a Bosh Ops File. In below example we're configuring containerLogMaxSize equal to 100Mi.
   

   $ cat ops.yaml

   - type: replace
     path: /instance_groups/name=worker/jobs/name=kubelet/properties/kubelet-configuration/containerLogMaxSize?
     value: 100Mi 

   3. Deploy the changes:
   $ bosh -d service-instance_<xxxx> deploy service-instance_<xxxx>.manifest -o ops.yaml 
   
   Please note the above change is persistent across node recreation, but not persistent across TKGi upgrades, as new default manifests will be generated and applied.
   
   
   

2. Using TKGi Kubernetes Profiles to pass kubelet CLI flags (deprecated):

The steps below allow you to configure container-log-max-size and container-log-max-files passed to kubelet as command flags instead of through the kubelet config file.
This method is deprecated and you'll find messages like the one below in /var/vcap/sys/log/kubelet/kubelet.stderr.log:

Flag --container-log-max-size has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.

To create experimental customized K8s Profile, please refer to the official Docs: https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid-Integrated-Edition/1.19/tkgi/GUID-k8s-profiles.html

 Please note: Experimental customizations are NOT validated or supported by TKGi team (same would be mentioned in above document)

1. 1. Create k8s profile ("container-log-max-size": "500Mi" as an example below).
      

      {
          "name":"kubelet-log",
          "description":"kubelet profile",
          "experimental_customizations":[
              {
                  "component":"kubelet",
                  "arguments":{
                      "container-log-max-size": "500Mi"
                  }
              }
          ]
       
      }

   2. tkgi create-kubernetes-profile <kubernetes-profile-JSON-path>
   3. tkgi update-cluster <Cluster-Name> --kubernetes-profile <Kubernetes-Profile-Name> ----> ** This will trigger changes in the cluster 
   4. To validate the changes inside the node:
      # ps -elf | grep kubelet

      E.g:
      # ps -elf | grep kubelet
4 S root      461105  461054  4  70 -10 - 506263 futex_ 13:09 ?       00:02:01 kubelet --container-log-max-size=500Mi 

N.B:  container-log-max-files parameter can be added like wise container-log-max-size in the K8s Profile.