SaltStack RAAS is unreachable with a failed to connect to storage backend error. Postgresql service is not running due to full partition.

Products

• VCF Automation

Issue/Introduction

• The entire SaltStack environment becomes completely unresponsive, and the RAAS service is unreachable. Upon checking the services, the RAAS service is in a failed state and the following critical error is observed in the RAAS service logs:

• • [CRITICAL] Failed to connect to the storage backend. Trying again in a bit

• Postgres service is down due to a full disk partition.

Environment

• SaltStack / VMware Aria Automation Config
• Red Hat Enterprise Linux (RHEL) 8
• PostgreSQL (RAAS DB)

Cause

The database partition running the RAAS database has exhausted all available free disk space. This storage exhaustion prevents the RAAS service from successfully connecting to the database backend, causing the service to fail.

Resolution

To resolve this issue, free up space on the database partition and restart the services:

 

 

1. Log in to the RAAS database server (Execute the default connection command (you will be prompted for the postgres user password):

   psql -U postgres -h localhost

2. List All Available Databases and locate your target database name (e.g., raas_xxxxx) from the displayed list.

   \l

3. Connect to the Target Database (Switch directly to your target database from within the interactive prompt)

   \c raas_xxxxx

4. Increase the Volume Group (VG) and expand the Logical Volume (LVM) for the DB partition to provide sufficient capacity (e.g., expand to 150GB).
5. Perform a vacuum full operation on the RAAS database to reclaim unused disk space.
6. Verify that the database footprint is reduced and that there is now plenty of free space on the partition.
7. Verify that the RAAS database auto-trim feature is configured correctly (e.g., set to 90 days) to help manage ongoing space utilization and prevent future occurrence using steps from KB:
   • Trimming the Aria Automation Config Postgres SQL database to increase UI performance
     https://knowledge.broadcom.com/external/article?articleNumber=370443
8. Bring the database services back online.
9. Restart the RAAS service and verify that it successfully connects to the storage backend.