PostGres VM disk filling up in Tanzu Hub

Products

• VMware Tanzu Platform Core

Issue/Introduction

The Tanzu Hub PostgreSQL VM reports disk usage at or approaching 100%, potentially leading to service instability or database "Read-Only" states. Users may observe the following:

- Log Bloat: Manual inspection shows that the /data/log directory is significantly larger than the /data/base directory.

- High I/O Wait: CPU performance metrics show I/O Wait times exceeding 10% (Normal is <5%). In critical cases, this may reach 25-30%.

- Performance Degradation: Slow query execution and application latency due to disk saturation.

Run the following to identify the distribution of disk usage:

bosh -d <hub-deployment-ID> ssh postgres -c "du -h --max-depth=2 /var/vcap/store/pvc-*"

Environment

Tanzu Hub 10.x

Cause

Undersized VM profiles or slow storage backends lead to high I/O Wait. This slows down queries, which in turn generates more "Slow Query" log entries, further saturating the disk.

Resolution

Upgrade Tanzu Hub to version 10.3.7 or 10.4.0. These versions include specific optimization features designed to prevent log-driven disk exhaustion.

If upgrade is not possible, as a workaround: 

Delete only files ending in .log within the /data/log directory.

CRITICAL SAFETY NOTE

Do not delete WAL files (found in /pg_wal) or any files in the /data/base directory. Deleting these will result in permanent data loss.

If I/O Wait remains above 5% after the log cleanup, the infrastructure is likely under-provisioned for the current application scale. Upsize the DB VM resource profile to provide higher IOPS