How to Enable Audit Logging in VMware vSphere with Tanzu Environments

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

^{Customers running VMware vSphere with Tanzu environment may require audit logging to track API server activity for compliance, troubleshooting, or security monitoring. The Audit logs are enabled by default for the Supervisor and the workload clusters that it deploys.}

^{The purpose of this article is to provide guidance on enabling and configuring Kubernetes audit logging in vSphere with Tanzu Supervisor and Tanzu Kubernetes Clusters (TKCs), aligned with Broadcom’s supported practices.}

Cause

^{The Kubernetes API server requires explicit configuration to capture and persist audit events. Without audit policies and log targets, critical information such as user authentication, resource modifications, and API calls may not be recorded. In regulated environments, this gap can lead to compliance risks and reduced operational visibility.}

Resolution

^{1. Understand Audit Logging Components}

^{Audit logging in Kubernetes consists of:}

^{Audit Policy: Defines which events are logged and at what level (Metadata, Request, RequestResponse).}

^{Audit Backend: Determines where logs are written (e.g., files, webhook).}

^{Audit Log Files: Stored on the API server node for review and forwarding to external systems (e.g., ELK, Splunk).}

^{2. Configure Audit Logging in vSphere with Tanzu}

^{Follow the supported procedure outlined in Broadcom TechDocs: 🔗 https://techdocs.broadcom.com/us/en/vmware-tanzu/standalone-components/tanzu-kubernetes-grid/2-4/using-tkg/workload-security-audit-log.html}

^{Key steps include:}

^{Create or update an audit policy file (e.g., audit-policy.yaml).}

^{Mount the audit policy file into the Kubernetes API server pod.}

^{Configure the API server startup parameters to reference the audit policy and log file location.}

^{Example snippet:}

^{apiVersion: audit.k8s.io/v1}

^{kind: Policy}

^rules:

  - level: Metadata

    verbs: ["create", "update", "patch", "delete"]

    resources: ["pods", "deployments", "services"]

^{3. Validate Audit Logging}

^{Access the API server node and check the audit log file (commonly /var/log/kubernetes/audit.log).}

^{Perform test operations (e.g., create/delete a pod) and confirm entries are logged.}

^{Example log entry:}

^{{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","verb":"create","user":"[email protected]","objectRef":{"resource":"pods","name":"nginx"}}}

^{4. Forward Logs to External Systems (Optional)}

^{For enterprise observability and compliance:}

^{Integrate audit logs with ELK/EFK stack, Splunk, or SIEM tools.}

^{Use Kubernetes log shipping agents (e.g., Fluent Bit, Filebeat) to forward logs securely.}

^{Support Constraints & Recommendations:}

^{Audit logging configuration must be applied at the cluster level; Supervisor and TKCs require separate configuration.}

^{Ensure sufficient disk space and log rotation policies to prevent node resource exhaustion.}

^{Use minimal logging levels in production to balance visibility and performance.}

Additional Information

^{https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/} 

^{https://techdocs.broadcom.com/us/en/vmware-tanzu/standalone-components/tanzu-kubernetes-grid/2-4/using-tkg/workload-security-audit-log.html}