Unable to run Postgres VACUUM due to "connection to server on socket failed"

Products

VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

When attempting to resolve a 100% full root partition (/dev/mapper/system-system_0) in Aria Suite Lifecycle by following Broadcom KB Unable to Log into Lifecycle Manager Due to Drive Space Issue, administrators may encounter the following roadblocks that prevent clearing the database bloat:

When attempting to switch to the postgres user (su - postgres), the system throws a warning:

"Your account has expired; please contact your system administrator. su: Authentication token expired (Ignored)."

When attempting to connect to the database to run the VACUUM command

```
./psql -d vrlcm
```

It fails with the following error:

"psql.bin: error: connection to server on socket "<directory>" failed: No such file or directory"

When accessing the Aria Lifecycle UI, the following message is presented:
- ```
Waiting for services to start
```

Note: This KB is a supplement to the KB Unable to Log into Lifecycle Manager Due to Drive Space Issue. Thus, it's recommended to follow it before proceeding with this KB.

Environment

VMware Aria Suite Lifecycle (LCM) 8.18.x

Cause

This issue is a "Catch-22" scenario caused by absolute disk exhaustion:

The Socket Error: The PostgreSQL service (vpostgres) crashes when the disk hits exactly 100% capacity because it requires a small amount of free space to create temporary lock files. Because the database service is down, administrators cannot connect to it to run the VACUUM command required to clear the space.

Resolution

To break the cycle, it's a must to blindly clear temporary log and cache files as root to carve out enough space to start the PostgreSQL service. Then, run the vacuum command via sudo to bypass the expired account shell.

Free up temporary space

Log into the LCM appliance via SSH as root and run the following commands sequentially to clear package caches and system logs:
```
tdnf clean all journalctl --vacuum-time=1d truncate -s 0 /var/log/messages
```
Run df -h and verify that /dev/mapper/system-system_0 has dropped to 99% or shows at least a few megabytes of available space.

Start the PostgreSQL Service

Now that the disk has a minimal amount of free space, start the database service:
```
systemctl restart vpostgres
```
Verify the service is running (press q to exit the status screen):
```
systemctl status vpostgres
```

Run the VACUUM command (Bypassing su)

Execute the database vacuum command directly from the root prompt. This bypasses the expired authentication token issue by utilizing the following:
```
sudo -u postgres /opt/vmware/vpostgres/current/bin/psql -d vrlcm -c "VACUUM FULL verbose analyze;"
```

Note: This process may take several minutes to complete depending on the size of the database bloat.

Verify Space and Restart LCM Services

Once the vacuum completes, verify that a healthy amount of disk space has been reclaimed (the partition typically drops into the 70% range):
```
df -h
```
Finally, restart the main LCM server service to restore web UI access:
```
systemctl restart vrlcm-server
```

Note: It may take 3 to 5 minutes for the Aria Suite Lifecycle UI to fully initialize and become accessible in a web browser.

Additional Information

Regarding the issue when attempting to switch to the postgres user (su - postgres), the system throws a warning:

"Your account has expired; please contact your system administrator. su: Authentication token expired (Ignored)."

To resolve the issue, follow the KB "Authentication token expired" error when switching to postgres user