Clustered database service will not quit unexpectedly and will not start

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Your user interface for RaaS is not receiving updates and 

[root@########### ~]# systemctl status postgresql-13.service

● postgresql-13.service - PostgreSQL 13 database server

   Loaded: loaded (/usr/lib/systemd/system/postgresql-13.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Tue 2025-06-24 19:25:47 UTC; 22h ago

     Docs: https://www.postgresql.org/docs/13/static/

  Process: 5699 ExecStart=/usr/pgsql-13/bin/postmaster -D ${PGDATA} (code=exited, status=1/FAILURE)

  Process: 5693 ExecStartPre=/usr/pgsql-13/bin/postgresql-13-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)

Main PID: 5699 (code=exited, status=1/FAILURE)

Environment

Aria Config - all versions

Postgresql - 13.x

Tanzu Salt - all versions

This applies only to environments that use streaming replication for the Postgresql database

 

Cause

The database server has likely run out of disk space due with the largest amount of space used in the pg_wal directory. This indicates that the primary database server is not in synchronous_commit mode and the replicators have not been syncing data. By default, the primary database server will store transactions to be synchronized forever, or until the replicator comes back online and begins to sync transactions again.

Find the pg_wal directory with this command (normally at /var/lib/postgres/13/data/pg_wal) :

find / -type d -name "pg_wal" 

 

Resolution

If the disk is full on the primary database server, then you will need to find some unnecessary files that can be removed. Old log files are usually a good place to start before deleting other files. You may try to download the files prior to removing them from the server in case you'd like to perform further investigation on your own. 

Once you have cleared enough space that the output from "df -h" does not report 100% used on the partition in use by the database, then try starting the database service with systemctl start postgresql-13. Once the database has started successfully (confirm with systemctl status postgresql-13), then do the same on the replicator server(s). 

Monitor the pg_wal directory for data to begin to be removed. This may take 10 - 15 minutes depending on overall architecture performance.