Tanzu Hub PostgreSQL WAL file growth leads to insufficient disk space

Products

• VMware Tanzu for Postgres
• Tanzu Hub SaaS
• VMware Tanzu Platform - Hub

Issue/Introduction

• Postgres filesystem is above the 80% capacity threshold.
• This can be checked using the following commands from Ops Manager or any system with access to the BOSH instance that Tanzu Hub is deployed on:

bosh instances #list instances to check registry deployment details

bosh ssh registry -d <DEPLOYMENT> #use deployment from previous command

kubectl exec -n tanzusm -it -c pg-container postgresql-0 -- df -h "/pgsql"

Environment

Tanzu Hub 10.2

Cause

For Tanzu Hub deployments with high PostgreSQL load it is possible that WAL (write ahead logs) may grow and lead to insufficient disk space for the PostgreSQL instance. This will result in Tanzu Hub unavailability. For deployments using the enterprise profile, data integrity issues may occur due to the high availability configuration of Postgres where WAL files are used for read replica synchronization.

Resolution

The scheduled PostgreSQL backups should be enabled as each backup archives all WAL files up to the point of the backup and includes them in the backup content which enables the system to automatically delete them afterwards and free up the consumed storage. For 10.2.* versions of Tanzu Hub, a script can be provided to enable these backups. Please contact your account team for details on the script, which includes both the script and instructions for execution.

This issue can still occur in 10.3 release of Tanzu Hub. Please follow Tanzu Hub release notes known issues and raise a support ticket if you require assistance with resolution.