Enable or modify kubernetes audit logging on already deployed workload cluster.
search cancel

Enable or modify kubernetes audit logging on already deployed workload cluster.

book

Article ID: 313111

calendar_today

Updated On:

Products

VMware VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:
Kubernetes audit logs record requests to the Kubernetes API server. To enable Kubernetes auditing on a Management or Tanzu Kubernetes cluster, set the ENABLE_AUDIT_LOGGING variable to true before you deploy the cluster.

Environment

Tanzu Kubernetes Grid 1.6.0

Cause

For clusters deployed with "ENABLE_AUDIT_LOGGING" flag set to true, If you want modify existing audit policy under /etc/kubernetes/audit-policy.yaml, direct changes are ephemeral and if the VM is recreated it will be overwritten.

Resolution

Clusters deployed with "ENABLE_AUDIT_LOGGING" flag set to false or deployed with "ENABLE_AUDIT_LOGGING" flag set to true however, modification on existing audit-policy.yaml is needed, can have Kubernetes audit logging enabled manually or modifying existing audit policy by modifying the kubeadmcontrolplane of this cluster as per the below procedure:
NOTE: This procedure applies to Legacy cluster only, not ClusterClass

1) Collect the KCP for the relevant cluster: 
kubectl get kcp test-control-plane -o yaml > test-control-plane.yaml
2) A - Edit the KCP to add the following to the 'apiServer' section: 
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          audit-log-maxage: "30"
          audit-log-maxbackup: "10"
          audit-log-maxsize: "100"
          audit-log-path: /var/log/kubernetes/audit.log
          audit-policy-file: /etc/kubernetes/audit-policy.yaml        
        extraVolumes: 
        - name: audit-policy 
          hostPath: /etc/kubernetes/audit-policy.yaml 
          mountPath: /etc/kubernetes/audit-policy.yaml 
        - name: audit-logs 
          hostPath: /var/log/kubernetes 
          mountPath: /var/log/kubernetes
B - and add the following to the 'files' section: 
spec:
  kubeadmConfigSpec:
    files:
    - content: YXBpVmVyc2lvbjogYXVkaXQuazhzLmlvL3YxYmV0YTEKa2luZDogUG9saWN5CnJ1bGVzOgogICMhIFRoZSBmb2xsb3dpbmcgcmVxdWVzdHMgd2VyZSBtYW51YWxseSBpZGVudGlmaWVkIGFzIGhpZ2gtdm9sdW1lIGFuZCBsb3ctcmlzaywKICAjISBzbyBkcm9wIHRoZW0uCiAgLSBsZXZlbDogTm9uZQogICAgdXNlcnM6IFsic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmt1YmUtcHJveHkiXQogICAgdmVyYnM6IFsid2F0Y2giXQogICAgcmVzb3VyY2VzOgogICAgICAtIGdyb3VwOiAiIiAjISBjb3JlCiAgICAgICAgcmVzb3VyY2VzOiBbImVuZHBvaW50cyIsICJzZXJ2aWNlcyIsICJzZXJ2aWNlcy9zdGF0dXMiXQogIC0gbGV2ZWw6IE5vbmUKICAgIHVzZXJHcm91cHM6IFsic3lzdGVtOm5vZGVzIl0KICAgIHZlcmJzOiBbImdldCJdCiAgICByZXNvdXJjZXM6CiAgICAgIC0gZ3JvdXA6ICIiICMhIGNvcmUKICAgICAgICByZXNvdXJjZXM6IFsibm9kZXMiLCAibm9kZXMvc3RhdHVzIl0KICAtIGxldmVsOiBOb25lCiAgICB1c2VyczoKICAgICAgLSBzeXN0ZW06a3ViZS1jb250cm9sbGVyLW1hbmFnZXIKICAgICAgLSBzeXN0ZW06a3ViZS1zY2hlZHVsZXIKICAgICAgLSBzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06ZW5kcG9pbnQtY29udHJvbGxlcgogICAgdmVyYnM6IFsiZ2V0IiwgInVwZGF0ZSJdCiAgICBuYW1lc3BhY2VzOiBbImt1YmUtc3lzdGVtIl0KICAgIHJlc291cmNlczoKICAgICAgLSBncm91cDogIiIgIyEgY29yZQogICAgICAgIHJlc291cmNlczogWyJlbmRwb2ludHMiXQogIC0gbGV2ZWw6IE5vbmUKICAgIHVzZXJzOiBbInN5c3RlbTphcGlzZXJ2ZXIiXQogICAgdmVyYnM6IFsiZ2V0Il0KICAgIHJlc291cmNlczoKICAgICAgLSBncm91cDogIiIgIyEgY29yZQogICAgICAgIHJlc291cmNlczogWyJuYW1lc3BhY2VzIiwgIm5hbWVzcGFjZXMvc3RhdHVzIiwgIm5hbWVzcGFjZXMvZmluYWxpemUiXQogICMhIERvbid0IGxvZyBIUEEgZmV0Y2hpbmcgbWV0cmljcy4KICAtIGxldmVsOiBOb25lCiAgICB1c2VyczoKICAgICAgLSBzeXN0ZW06a3ViZS1jb250cm9sbGVyLW1hbmFnZXIKICAgIHZlcmJzOiBbImdldCIsICJsaXN0Il0KICAgIHJlc291cmNlczoKICAgICAgLSBncm91cDogIm1ldHJpY3MuazhzLmlvIgogICMhIERvbid0IGxvZyB0aGVzZSByZWFkLW9ubHkgVVJMcy4KICAtIGxldmVsOiBOb25lCiAgICBub25SZXNvdXJjZVVSTHM6CiAgICAgIC0gL2hlYWx0aHoqCiAgICAgIC0gL3ZlcnNpb24KICAgICAgLSAvc3dhZ2dlcioKICAjISBEb24ndCBsb2cgZXZlbnRzIHJlcXVlc3RzLgogIC0gbGV2ZWw6IE5vbmUKICAgIHJlc291cmNlczoKICAgICAgLSBncm91cDogIiIgIyEgY29yZQogICAgICAgIHJlc291cmNlczogWyJldmVudHMiXQogICMhIERvbid0IGxvZyBUTUMgc2VydmljZSBhY2NvdW50IHBlcmZvcm1pbmcgcmVhZCBvcGVyYXRpb25zIGJlY2F1c2UgdGhleSBhcmUgaGlnaC12b2x1bWUuCiAgLSBsZXZlbDogTm9uZQogICAgdXNlckdyb3VwczogWyJzeXN0ZW06c2VydmljZWFjY291bnRzOnZtd2FyZS1zeXN0ZW0tdG1jIl0KICAgIHZlcmJzOiBbImdldCIsICJsaXN0IiwgIndhdGNoIl0KICAjISBEb24ndCBsb2cgcmVhZCByZXF1ZXN0cyBmcm9tIGdhcmJhZ2UgY29sbGVjdG9yIGJlY2F1c2UgdGhleSBhcmUgaGlnaC12b2x1bWUuCiAgLSBsZXZlbDogTm9uZQogICAgdXNlcnM6IFsic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtOmdlbmVyaWMtZ2FyYmFnZS1jb2xsZWN0b3IiXQogICAgdmVyYnM6IFsiZ2V0IiwgImxpc3QiLCAid2F0Y2giXQogICMhIG5vZGUgYW5kIHBvZCBzdGF0dXMgY2FsbHMgZnJvbSBub2RlcyBhcmUgaGlnaC12b2x1bWUgYW5kIGNhbiBiZSBsYXJnZSwgZG9uJ3QgbG9nIHJlc3BvbnNlcyBmb3IgZXhwZWN0ZWQgdXBkYXRlcyBmcm9tIG5vZGVzCiAgLSBsZXZlbDogUmVxdWVzdAogICAgdXNlckdyb3VwczogWyJzeXN0ZW06bm9kZXMiXQogICAgdmVyYnM6IFsidXBkYXRlIiwicGF0Y2giXQogICAgcmVzb3VyY2VzOgogICAgICAtIGdyb3VwOiAiIiAjISBjb3JlCiAgICAgICAgcmVzb3VyY2VzOiBbIm5vZGVzL3N0YXR1cyIsICJwb2RzL3N0YXR1cyJdCiAgICBvbWl0U3RhZ2VzOgogICAgICAtICJSZXF1ZXN0UmVjZWl2ZWQiCiAgIyEgZGVsZXRlY29sbGVjdGlvbiBjYWxscyBjYW4gYmUgbGFyZ2UsIGRvbid0IGxvZyByZXNwb25zZXMgZm9yIGV4cGVjdGVkIG5hbWVzcGFjZSBkZWxldGlvbnMKICAtIGxldmVsOiBSZXF1ZXN0CiAgICB1c2VyczogWyJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06bmFtZXNwYWNlLWNvbnRyb2xsZXIiXQogICAgdmVyYnM6IFsiZGVsZXRlY29sbGVjdGlvbiJdCiAgICBvbWl0U3RhZ2VzOgogICAgICAtICJSZXF1ZXN0UmVjZWl2ZWQiCiAgIyEgU2VjcmV0cywgQ29uZmlnTWFwcywgYW5kIFRva2VuUmV2aWV3cyBjYW4gY29udGFpbiBzZW5zaXRpdmUgJiBiaW5hcnkgZGF0YSwKICAjISBzbyBvbmx5IGxvZyBhdCB0aGUgTWV0YWRhdGEgbGV2ZWwuCiAgLSBsZXZlbDogTWV0YWRhdGEKICAgIHJlc291cmNlczoKICAgICAgLSBncm91cDogIiIgIyEgY29yZQogICAgICAgIHJlc291cmNlczogWyJzZWNyZXRzIiwgImNvbmZpZ21hcHMiXQogICAgICAtIGdyb3VwOiBhdXRoZW50aWNhdGlvbi5rOHMuaW8KICAgICAgICByZXNvdXJjZXM6IFsidG9rZW5yZXZpZXdzIl0KICAgIG9taXRTdGFnZXM6CiAgICAgIC0gIlJlcXVlc3RSZWNlaXZlZCIKICAjISBHZXQgcmVwc29uc2VzIGNhbiBiZSBsYXJnZTsgc2tpcCB0aGVtLgogIC0gbGV2ZWw6IFJlcXVlc3QKICAgIHZlcmJzOiBbImdldCIsICJsaXN0IiwgIndhdGNoIl0KICAgIHJlc291cmNlczoKICAgICAgLSBncm91cDogIiIgIyEgY29yZQogICAgICAtIGdyb3VwOiAiYWRtaXNzaW9ucmVnaXN0cmF0aW9uLms4cy5pbyIKICAgICAgLSBncm91cDogImFwaWV4dGVuc2lvbnMuazhzLmlvIgogICAgICAtIGdyb3VwOiAiYXBpcmVnaXN0cmF0aW9uLms4cy5pbyIKICAgICAgLSBncm91cDogImFwcHMiCiAgICAgIC0gZ3JvdXA6ICJhdXRoZW50aWNhdGlvbi5rOHMuaW8iCiAgICAgIC0gZ3JvdXA6ICJhdXRob3JpemF0aW9uLms4cy5pbyIKICAgICAgLSBncm91cDogImF1dG9zY2FsaW5nIgogICAgICAtIGdyb3VwOiAiYmF0Y2giCiAgICAgIC0gZ3JvdXA6ICJjZXJ0aWZpY2F0ZXMuazhzLmlvIgogICAgICAtIGdyb3VwOiAiZXh0ZW5zaW9ucyIKICAgICAgLSBncm91cDogIm1ldHJpY3MuazhzLmlvIgogICAgICAtIGdyb3VwOiAibmV0d29ya2luZy5rOHMuaW8iCiAgICAgIC0gZ3JvdXA6ICJwb2xpY3kiCiAgICAgIC0gZ3JvdXA6ICJyYmFjLmF1dGhvcml6YXRpb24uazhzLmlvIgogICAgICAtIGdyb3VwOiAic2V0dGluZ3MuazhzLmlvIgogICAgICAtIGdyb3VwOiAic3RvcmFnZS5rOHMuaW8iCiAgICBvbWl0U3RhZ2VzOgogICAgICAtICJSZXF1ZXN0UmVjZWl2ZWQiCiAgIyEgRGVmYXVsdCBsZXZlbCBmb3Iga25vd24gQVBJcwogIC0gbGV2ZWw6IFJlcXVlc3RSZXNwb25zZQogICAgcmVzb3VyY2VzOgogICAgICAtIGdyb3VwOiAiIiAjISBjb3JlCiAgICAgIC0gZ3JvdXA6ICJhZG1pc3Npb25yZWdpc3RyYXRpb24uazhzLmlvIgogICAgICAtIGdyb3VwOiAiYXBpZXh0ZW5zaW9ucy5rOHMuaW8iCiAgICAgIC0gZ3JvdXA6ICJhcGlyZWdpc3RyYXRpb24uazhzLmlvIgogICAgICAtIGdyb3VwOiAiYXBwcyIKICAgICAgLSBncm91cDogImF1dGhlbnRpY2F0aW9uLms4cy5pbyIKICAgICAgLSBncm91cDogImF1dGhvcml6YXRpb24uazhzLmlvIgogICAgICAtIGdyb3VwOiAiYXV0b3NjYWxpbmciCiAgICAgIC0gZ3JvdXA6ICJiYXRjaCIKICAgICAgLSBncm91cDogImNlcnRpZmljYXRlcy5rOHMuaW8iCiAgICAgIC0gZ3JvdXA6ICJleHRlbnNpb25zIgogICAgICAtIGdyb3VwOiAibWV0cmljcy5rOHMuaW8iCiAgICAgIC0gZ3JvdXA6ICJuZXR3b3JraW5nLms4cy5pbyIKICAgICAgLSBncm91cDogInBvbGljeSIKICAgICAgLSBncm91cDogInJiYWMuYXV0aG9yaXphdGlvbi5rOHMuaW8iCiAgICAgIC0gZ3JvdXA6ICJzZXR0aW5ncy5rOHMuaW8iCiAgICAgIC0gZ3JvdXA6ICJzdG9yYWdlLms4cy5pbyIKICAgIG9taXRTdGFnZXM6CiAgICAgIC0gIlJlcXVlc3RSZWNlaXZlZCIKICAjISBEZWZhdWx0IGxldmVsIGZvciBhbGwgb3RoZXIgcmVxdWVzdHMuCiAgLSBsZXZlbDogTWV0YWRhdGEKICAgIG9taXRTdGFnZXM6CiAgICAgIC0gIlJlcXVlc3RSZWNlaXZlZCIK
      path: /etc/kubernetes/audit-policy.yaml
      owner: "root:root"
      permissions: "0600"
      encoding: base64
Note that 'content' is all one line and that is actually just the following yaml converted to base64 via the below command: (please review and modify accordingly) 
cat audit-policy.yaml | base64 -w 0
audit-policy.yaml 
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
  #! The following requests were manually identified as high-volume and low-risk,
  #! so drop them.
  - level: None
    users: ["system:serviceaccount:kube-system:kube-proxy"]
    verbs: ["watch"]
    resources:
      - group: "" #! core
        resources: ["endpoints", "services", "services/status"]
  - level: None
    userGroups: ["system:nodes"]
    verbs: ["get"]
    resources:
      - group: "" #! core
        resources: ["nodes", "nodes/status"]
  - level: None
    users:
      - system:kube-controller-manager
      - system:kube-scheduler
      - system:serviceaccount:kube-system:endpoint-controller
    verbs: ["get", "update"]
    namespaces: ["kube-system"]
    resources:
      - group: "" #! core
        resources: ["endpoints"]
  - level: None
    users: ["system:apiserver"]
    verbs: ["get"]
    resources:
      - group: "" #! core
        resources: ["namespaces", "namespaces/status", "namespaces/finalize"]
  #! Don't log HPA fetching metrics.
  - level: None
    users:
      - system:kube-controller-manager
    verbs: ["get", "list"]
    resources:
      - group: "metrics.k8s.io"
  #! Don't log these read-only URLs.
  - level: None
    nonResourceURLs:
      - /healthz*
      - /version
      - /swagger*
  #! Don't log events requests.
  - level: None
    resources:
      - group: "" #! core
        resources: ["events"]
  #! Don't log TMC service account performing read operations because they are high-volume.
  - level: None
    userGroups: ["system:serviceaccounts:vmware-system-tmc"]
    verbs: ["get", "list", "watch"]
  #! Don't log read requests from garbage collector because they are high-volume.
  - level: None
    users: ["system:serviceaccount:kube-system:generic-garbage-collector"]
    verbs: ["get", "list", "watch"]
  #! node and pod status calls from nodes are high-volume and can be large, don't log responses for expected updates from nodes
  - level: Request
    userGroups: ["system:nodes"]
    verbs: ["update","patch"]
    resources:
      - group: "" #! core
        resources: ["nodes/status", "pods/status"]
    omitStages:
      - "RequestReceived"
  #! deletecollection calls can be large, don't log responses for expected namespace deletions
  - level: Request
    users: ["system:serviceaccount:kube-system:namespace-controller"]
    verbs: ["deletecollection"]
    omitStages:
      - "RequestReceived"
  #! Secrets, ConfigMaps, and TokenReviews can contain sensitive & binary data,
  #! so only log at the Metadata level.
  - level: Metadata
    resources:
      - group: "" #! core
        resources: ["secrets", "configmaps"]
      - group: authentication.k8s.io
        resources: ["tokenreviews"]
    omitStages:
      - "RequestReceived"
  #! Get repsonses can be large; skip them.
  - level: Request
    verbs: ["get", "list", "watch"]
    resources:
      - group: "" #! core
      - group: "admissionregistration.k8s.io"
      - group: "apiextensions.k8s.io"
      - group: "apiregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "metrics.k8s.io"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
    omitStages:
      - "RequestReceived"
  #! Default level for known APIs
  - level: RequestResponse
    resources:
      - group: "" #! core
      - group: "admissionregistration.k8s.io"
      - group: "apiextensions.k8s.io"
      - group: "apiregistration.k8s.io"
      - group: "apps"
      - group: "authentication.k8s.io"
      - group: "authorization.k8s.io"
      - group: "autoscaling"
      - group: "batch"
      - group: "certificates.k8s.io"
      - group: "extensions"
      - group: "metrics.k8s.io"
      - group: "networking.k8s.io"
      - group: "policy"
      - group: "rbac.authorization.k8s.io"
      - group: "settings.k8s.io"
      - group: "storage.k8s.io"
    omitStages:
      - "RequestReceived"
  #! Default level for all other requests.
  - level: Metadata
    omitStages:
      - "RequestReceived"
Once you finish editing the KCP, new control plane nodes will be rolled out gracefully for that cluster.

Powered by Wolken Software