tenant-manager pods in CrashLoopBackOff due to bloated audit_event table in VCF Automation

Products

• VCF Automation

Issue/Introduction

• VCF Automation portal returns "no healthy upstream" error
• tenant-manager pods are stuck in CrashLoopBackOff state
• Postgres database partition reaches 100% utilization
• The audit_trail table is excessively large (e.g., >15GB)

Environment

VCF Automation 9.0.x

Cause

The issue is caused by a defect in the background scheduler responsible for cleaning up old audit records.

In certain corner cases - specifically when a restart or failover occurs within 60 seconds of a scheduled purge the purger job can get stuck in an indefinite wait state.
Once stuck, the job is never resubmitted, leading to a massive accumulation of audit data. This bloat eventually fills the database partition, causing the database to lose its leader and the tenant-manager pods to crash.

Resolution

A permanent fix for this condition is under development by VMware Engineering and will be included in an upcoming version, subscribe to this article to be informed once a solution is available.

To validate the purger job is stuck:

1. Connect to the VCFA appliance shell session as vmware-system-user, refer to KB 436828
2. Run the below command to review the scheduler job status:

   k -n prelude exec -it statefulset.apps/vcfapostgres -- psql -U postgres -d tenantmanager -c "SELECT id, client_activity_name, last_execution_time, next_execution_time FROM scheduled_activity_jobs WHERE client_activity_name LIKE '%ActivityLogCleanerJobActivity%';"

   If the next_execution_time is in the past or the job has not run recently despite significant data growth, the scheduler is likely stuck.

3. Open a Support Request: If the job is confirmed to be stuck or the environment is already failing, open a Support Request (SR) with Broadcom Support. Manual intervention at the database level is required to clear the accumulated data and restart the scheduler.

Workaround: If immediate database clean-up is not possible, then increase the storage using Resize Storage under Fleet Management > Lifecycle > Components to allow the services to start. This provides the necessary overhead for the environment to remain operational while Broadcom Support performs manual remediation.