Update TKGi kube-apiserver audit logs retention policy

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

kube-apiserver audit logs can be a useful resource to troubleshoot cluster issues as they record all the interactions of clients and services with the Kubernetes API.

In TKGi clusters, audit logs are stored by default inside the master nodes' "/var/vcap/sys/log/kube-apiserver/audit/log/" directory.

The default audit configuration, together with other kube-apiserver flags, can be viewed in "/var/vcap/data/jobs/kube-apiserver/<id>/config/bpm.yml" file.
At the time of writing this KB, the default audit logs retention policy looks as follows:

  - "--audit-log-maxage=30"

  - "--audit-log-maxbackup=10"

  - "--audit-log-maxsize=100"

  - "--audit-log-path=/var/vcap/sys/log/kube-apiserver/audit/log/audit.log"

  - "--audit-policy-file=/var/vcap/jobs/kube-apiserver/config/audit_policy.yml"

Details on audit policy flags can be checked in the Kubernetes official Docs, Log backend.

There may be situations in which you'd want to customize the audit logs retention policy, for example, allowing more files to be stored before rotation takes place, or increasing the size of each of the audit log files.

This KB walks you through how to achieve this configuration through TKGi Kubernetes Profiles.

Resolution

To configure the kube-apiserver audit logs retention policy you need to create a TKGi Kubernetes Profile and update your cluster with it.

Example of audit log retention Kubernetes Profile:

{  
    "name": "audit-logs",  
    "description": "Audit Logs Profile",   
    "experimental_customizations": [  
        {  
            "component": "kube-apiserver",  
            "arguments": {  
                "audit-log-maxbackup": "20",  
                "audit-log-maxsize": "200"
            }  
        }  
    ]  
}

In the above example we're indicating we want to keep up to 20 audit log files (audit-log-maxbackup), each of them with a max size of 200 MB (audit-log-maxsize). After an audit log file reaches 200 MB, it'll be rotated and a new one will be created in "/var/vcap/sys/log/kube-apiserver/audit/log/" directory.

The steps to configure your clusters with the above Kubernetes Profile are as follows:

New Cluster Creation

1. Create a k8s-profile json file with the above configuration:
   # vim audit-logs-k8s-profile.json
   
   
2. Create a k8s-profile based on the json file:
   # tkgi create-k8s-profile audit-logs-k8s-profile.json
   
   
3. List up all the k8s-profiles and confirm the audit-logs profile creation:
   # tkgi k8s-profiles
   
   
4. Create a new cluster using the k8s-profile:
   # tkgi create-cluster CLUSTER-NAME --external-hostname HOSTNAME --plan PLAN-NAME --kubernetes-profile audit-logs
   
   

Update existing Cluster without any k8s-profile applied to it

1. Create a k8s-profile json file with the above configuration:
   # vim audit-logs-k8s-profile.json
   
   
2. Create a k8s-profile based on the json file:
   # tkgi create-k8s-profile audit-logs-k8s-profile.json
   
   
3. List up all the k8s-profiles and confirm the audit-logs profile creation:
   # tkgi k8s-profiles
   
   
4. Update the cluster with the k8s-profile:
   # tkgi update-cluster CLUSTER-NAME --kubernetes-profile audit-logs
   
   

Update existing Cluster that already has a k8s-profile applied to it

If you're updating a cluster that already has a k8s-profile applied to it and you want to retain that configuration, you need to create a new k8s-profile with both configurations combined, i.e. the existing k8s-profile configuration plus the new audit-logs configuration. Then you update the cluster with the new k8s-profile.

1. Check which k8s-profile is already applied to your cluster:
   # tkgi cluster CLUSTER-NAME
   
   Check the Kubernetes Profile Name field. If nothing shows up there, it means your cluster doesn't have any k8s-profile applied.
   
   
2. Get the applied k8s-profile configuration:
   # tkgi k8s-profile <applied-k8s-profile-name> --json > new-combined-k8s-profile.json
   
   For example:
   
   
   {  
       "name": "my-profile3",
   "uuid": "<UUID>",
   "owner": "admin",
       "description": "My profile description",
       "created_at": 1749031948,
       "customizations": [  
           {  
               "component": "kube-apiserver",  
               "arguments": {  
                   "service-node-port-range": "30000-40000"  
               }  
           }  
       ],  
       "experimental_customizations": [  
           {  
               "component": "kubelet",  
               "arguments": {  
                   "maximum-dead-containers": "1000",  
                   "feature-gates": "APIListChunking=true"
               }  
           }  
       ]  
   }
   
   
3. Combine the existing k8s-profile configuration with the audit-logs one. Remove the "uuid", "owner", and "created_at" fields. Update the name with a unique one:
   # vim new-combined-k8s-profile.json
   
   For example:
   
   
   {  
       "name": "combined-k8s-profiles-audit-logs",
       "description": "My profile description",
       "customizations": [  
           {  
               "component": "kube-apiserver",  
               "arguments": {  
                   "service-node-port-range": "30000-40000"  
               }  
           }  
       ],  
       "experimental_customizations": [  
           {  
               "component": "kubelet",  
               "arguments": {  
                   "maximum-dead-containers": "1000",  
                   "feature-gates": "APIListChunking=true"
               }  
           },
           {
              "component": "kube-apiserver",
              "arguments": {
                 "audit-log-maxbackup": "20",
                 "audit-log-maxsize": "200"
              }
           }
       ]  
   } 
   
   
   
4. Create a new k8s-profile based on the json file:
   # tkgi create-k8s-profile new-combined-k8s-profile.json
   
   
5. List up all the k8s-profiles and confirm the combined-k8s-profiles-audit-logs profile creation:
   # tkgi k8s-profiles
   
   
6. Update the cluster with the k8s-profile:
   # tkgi update-cluster CLUSTER-NAME --kubernetes-profile combined-k8s-profiles-audit-logs