Gatekeeper-audit pod in CrashLoopBackOff due to OOMKilled status in TMC-SM managed clusters

Products

• VMware Tanzu Platform - Kubernetes

Issue/Introduction

In a production environment managed by Tanzu Mission Control Self-Managed (TMC-SM), the gatekeeper-audit pod may enter a CrashLoopBackOff state.

Symptoms include:

• Repeated pod restarts.
• Liveness and Readiness probe failures.
• Pod status showing Terminated with reason OOMKilled.

Cause

The root cause is the default memory limit (typically 512Mi) being insufficient for the volume of resource metadata in the cluster.

Resolution

To resolve this issue, the memory allocation for the gatekeeper-audit deployment must be increased.

1. Locate the gatekeeper-audit deployment in the gatekeeper-system namespace.
2. Update the container resource limits from 512Mi to higher (depending on cluster size).
   • Command-line example:
     kubectl edit deployment gatekeeper-audit -n <TMC NAMESPACE>
3. Monitor the pod using kubectl get pods -n <TMC NAMESPACE> to ensure it reaches a Running state and the restart count stabilizes.