Unable to Log In to SSP UI Due to PostgreSQL Disk Space Exhaustion and CrashLoopBackOff

Products

VMware vDefend Firewall VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Users are unable to log in to the Security Services Platform (SSP) user interface, receiving an incorrect username/password or account locked error.

Upon reviewing the Kubernetes cluster state in the nsxi-platform namespace, administrators will notice multiple cascading pod failures, specifically centered around the primary database:

The postgresql-ha-postgresql-0 pod is stuck in a CrashLoopBackOff status with a high restart count.
Dependent microservices (such as druid-coordinator, authserver, etc.) are stuck indefinitely in Init:0/X or CrashLoopBackOff states.
Checking the underlying storage reveals that the Persistent Volume Claim (PVC) for the PostgreSQL data volume (data-postgresql-ha-postgresql-0) has reached 100% consumption of its default 50Gi allocation.

An inspection of the host directory using du -sh will show the primary volume heavily consumed (e.g., ~47G+):

Execute the following commands in the SSP-Installer CLI:

Find the PVC UUID:
```
k get pvc -A | grep data-postgresql-ha-postgresql-0
```
Note down the pvc-<uuid> from the output to validate the directory size later.
Identify the Worker Node IP hosting PostgreSQL: Since the PostgreSQL pod might be in a CrashLoopBackOff, you can still find its assigned worker node by running:
```
k get pod postgresql-ha-postgresql-0 -n nsxi-platform -o wide
```
Look at the NODE column and note down the worker node's name (e.g., nsx-ssp-ab-0-####-abcd).
Retrieve the Node's Internal IP: Use the node name obtained from the previous step to get its IP address:
```
k get node <node-name-from-above> -o wide
```
Look at the INTERNAL-IP column. This is the <worker-node-ip> you will use to SSH into the machine.
Pro-Tip (One-Liner): To get the IP directly without traversing menus, run:
k get node $(k get pod postgresql-ha-postgresql-0 -n nsxi-platform -o jsonpath='{.spec.nodeName}') -o jsonpath='{.status.addresses[?(@.type=="InternalIP")].address}'

Validate Storage on the Worker Node: Log into the target worker node using the retrieved IP, elevate to root, and check the directory sizes:

ssh capv@<worker-node-ip>
sudo su -
du -sh /var/lib/kubelet/pods/*/volumes/kubernetes.io~csi/*/mount

Expected Output Example:

47G     /var/lib/kubelet/pods/088d13ed-846c-4495-a651-3cc29bced208/volumes/kubernetes.io~csi/pvc-<above-uuid>/mount <<<<<<<< OBSERVE THIS FIELD
90M     /var/lib/kubelet/pods/1a45d3a0-3581-4af9-91b3-cb7a1bef32c4/volumes/kubernetes.io~csi/pvc-uuid/mount
2.7G    /var/lib/kubelet/pods/3fec57be-62c1-4ddc-aa44-dac7db2bf646/volumes/kubernetes.io~csi/pvc-uuid/mount

Environment

vDefend Security Services Platform(SSP) version 5.1.1

Cause

The database disk space exhaustion occurs when the PostgreSQL automatic cleanup process (autovacuum) is blocked from reclaiming storage space.

In PostgreSQL, automatic maintenance cannot purge old or temporary data if a database transaction remains open. In this scenario, an internal background update or session became orphaned or stuck (potentially due to a transient network disruption or a temporary database lock conflict).

This orphaned session blocks routine database cleanup processes, leading to an accumulation of transaction logs and dead data until the default 50Gi storage limit was completely filled.

Resolution

This issue is resolved in the next release of SSP via two code-level enhancements:

Timeout Defenses: Background database processes have been updated to strictly adhere to default system guardrails (including a 1-hour statement timeout and 30-minute idle transaction timeout). This ensures any hung or orphaned background transactions are automatically terminated by the system.
Proactive Monitoring: A background monitoring service now automatically audits database health every 15 minutes. If a transaction begins to hang or block the cleanup window, the system will proactively log the event details directly into support bundles for immediate visibility.

The workaround exists. It is recommended to contact Broadcom Support before proceeding, as this recovery procedure involves direct modifications to the underlying PVC YAML configuration.

This is internally tracked through the ticket #3715709.