VCF Automation UI Error 500 Due to PostgreSQL Database Partition Full

Products

• VCF Automation

Issue/Introduction

Users are unable to access the VCF Automation (VCFA) interface, receiving an HTTP 500 error. The vcfa-health-verifier pods enter an Error state, and internal services, including the tenant manager, fail to initialize due to underlying database failures.

Pods in Error state, VCFA UI unavailable.

Environment

VCF Automation 9.1

Cause

The PostgreSQL pod partition has reached 100% capacity. This is triggered when the database size exceeds the allocated persistent volume claim (PVC) threshold, exacerbated by the accumulation of PostgreSQL Write Ahead Log (WAL) files.

Resolution

1. Resize the PostgreSQL PVC: Expand the PVC capacity for the PostgreSQL deployment. For example, increase the storage from 30Gi to 60Gi to accommodate the database and WAL logs.

2. Verify Pod Status: Confirm the PostgreSQL pod returns to a Running state and that the tenant manager service successfully initializes.

                    

   kubectl get pods -n 

    

3. Force Reconciliation of HelmReleases: To ensure all dependent services are correctly synchronized, trigger a reconciliation of the VCFA HelmReleases (addons, prelude deployer, and vksm). Execute the following command.
   
   kubectl annotate hr reconcile.fluxcd.io/requestedAt="$(date +%s)" --overwrite -n