SDDC database Leader election failures due to disk exhaustion in VCF Service Runtime 9.1

Products

• VCF Operations
• VMware SDDC Manager / VCF Installer

Issue/Introduction

Unable to perform Day 2 lifecycle management operations, specifically component upgrades, within the Fleet Management interface. The SDDC database in the VCF Service Runtime environment is non-functional as it lacks a leader postgres pod, preventing standard database operations.

Symptoms

• Cluster role status via: kubectl exec -it -n vcf-sddc-lcm vcf-sddc-lcm-db-0 -- patronictl list shows all pods in Replica role; no Leader present.

• Disk utilization check show 100% usage on the pgdata partition (Replace # with 0,1,2) :
  kubectl exec -it -n vcf-sddc-lcm vcf-sddc-lcm-db-# -- df -h

• Run du -Sh to identify high volume of accumulated WAL files inside the pg_wal folder in vcf-sddc-lcm-db pods (Replace # with 0,1,2) :  
  k exec -it -n vcf-sddc-lcm vcf-sddc-lcm-db-# -- du -Sh /home/postgres/pgdata/pgroot/data

Environment

VCF Operations 9.1

VCF SDDC 9.1

Cause

The database cluster service entered an unhealthy state due to disk exhaustion on the postgres pods. The accumulation of Write Ahead Logs (WAL) consumed all available storage on the pgdata partition, preventing the database cluster from electing a leader pod and resulting in connectivity failures for dependent services.

Resolution

To resolve the SDDC DB issue in VCF Service Runtime, create a support ticket with Broadcom Contact Broadcom support