VCF Automation UI Returns "Internal Server Error 500" Due to PostgreSQL TOAST Table Bloat Exhausting Disk Space

Products

• VCF Automation

Issue/Introduction

VMware Cloud Foundation (VCF) Automation is completely inaccessible, presenting an "Internal Server Error 500" in the user interface.

Investigation reveals that the underlying PostgreSQL database cluster is down because the master database pod (vcfapostgres-0) has reached 100% disk capacity. This total disk exhaustion prevents the master node from operating and forces the replica nodes (vcfapostgres-1 and vcfapostgres-2) into a start failed state. Additionally, automated cleanup jobs designed to purge old activity records are failing and crashing instantly.

Key Error Messages:

• Internal server error 500

• ERROR: Master pod disk usage (100%) exceeds threshold (95%).

• ERROR: Cannot remediate replicas when master disk is full.

Environment

 

• Product: VCF Automation (VCFA)

• Version: 9.x

• Component: Embedded vPostgres (tenantmanager database)

 

Cause

The database disk filled up to 100% capacity due to extreme storage bloat within the PostgreSQL TOAST tables, specifically within the activity partitions.

Continuous pod crashes and restarts in the environment generated massive database churn through frequent full-blob updates. For example, a single partition (activity_partition_15) grew to 6 GB despite containing only ~135 KB of actual data—a 99.9% bloat rate. Across the entire table, approximately 500 live rows consumed 20 GB of total disk space.

While the standard PostgreSQL autovacuum daemon was healthy and running, it only marks dead rows as available for future internal reuse; it does not return that freed space to the operating system. Because the TOAST data accumulated much faster than it could be reused, the physical file size rapidly ballooned until it exhausted all available disk space, crashing the cluster.

Resolution

To clear the bloat, reclaim disk space, and restore the VCF Automation UI, perform a manual cleanup of the bloated tables.

1. Stop VCF Automation Services Scale down or stop the VCF Automation application pods that actively interact with the database to prevent new writes during cleanup.

2. Access the Master Database Pod SSH into the appliance or access the Kubernetes cluster, then open a bash session inside the master PostgreSQL pod:

   Bash

    

   kubectl -n prelude exec -it vcfapostgres-0 -- /bin/bash
   su postgres
   psql
   \c tenantmanager
   

3. Execute the Cleanup Procedure Run the following SQL commands to delete the bloated dead rows. This will immediately drop the file sizes from gigabytes down to megabytes and reclaim the trapped disk space:

   SQL

    

   DELETE FROM activity;
   DELETE FROM scheduled_activity_jobs;
   DELETE FROM activity_pc_queue;
   DELETE FROM activity_pc_event_queue;
   DELETE FROM fifo_activity_queue;
   DELETE FROM task_activity_queue;
   DELETE FROM vc_activity_queue;
   

4. Restart VCF Automation Services Once the disk utilization drops safely below the 95% threshold, the master database pod will resume normal operations and synchronize with its replicas. Start the VCF Automation application pods to restore UI access.

    

    

5. Post-Maintenance Monitoring Monitor the disk growth over the next few days. If rapid bloat returns, execute a VACUUM FULL on the specific tables during a maintenance window to force compaction and return the trapped space directly to the operating system.