SSP Postgres Storage Exhausted Due to Slow Disk Speed and Flow Export Job Bloat

Products

• VMware vDefend Firewall
• VMware vDefend Firewall with Advanced Threat Prevention

Issue/Introduction

Users cannot log in to SSP due to Postgres disk storage reaching 100% capacity, forcing the database pod into a CrashLoopBackOff state.

Error:

Unable to login to SSP : Your login attempt was not successful. The username/password combination is incorrect or the account specified has been locked.

Environment

vDefend Security Services Platform(SSP) 5.1, 5.1.1

Environments with slower storage disks (high write latency) are particularly susceptible. Please note the shared storage latency requirement in System Requirements

Cause

This issue occurs due to a combination of a defect in the export job expiration logic and slow underlying storage infrastructure:

• Job Expiration Logic:
  
  A known issue prevents the permanent deletion of expired flow export jobs. Instead, the system periodically attempts to soft-delete these jobs.
  
  
• Postgres Storage Allocation:
  
  Postgres does not perform in-place updates. Every update writes an entirely new row and marks the old data as a "dead tuple" in the database TOAST table.
  
  
• Storage Latency Bottleneck:
  
  Postgres relies on a background auto-vacuum process to scan the disk and free up space taken by these dead tuples. In environments with slower storage, the auto-vacuum process cannot read and clear the dead tuples within the periodic job window.
  
  
• Storage Bloat:
  
  When the next job fires, it generates a new wave of dead tuples before the disk has finished cleaning up the previous ones. This race condition causes the storage to bloat rapidly, eventually filling the disk.

To confirm if your environment is experiencing this specific issue, generate both SSP (platform) and SSPI (installer) support bundles and perform the following checks:

• SSP UI Alarm:
  
  A Platform Disk Usage High alarm for postgresql-ha-postgresql-0 would have appeared before the disk filled up.
  
  
• PVC Usage:
  
  In the SSP support bundle, review var/log/ssp/supportbundle/kubectl_df_pv.txt to check the name, volume, and usage of the Postgres PVC (data-postgresql-ha-postgresql-0), and identify its worker node.
  
  
• Abnormal Storage Usage:
  
  In the SSP support bundle, review var/log/ssp/supportbundle/postgres_metrics/druid_postgres_sql_metrics. Locate the VACUUM_REPORT section. If intelligence.exportjob shows abnormally high storage usage (e.g., 40GB+), this issue is present.
  
  
• Identify Block Device:
  
  In the SSPI support bundle, review <worker node name>/system/mount to find the device corresponding to the PVC. Look for an entry similar to: /dev/sdh on /var/lib/containers/kubelet/pods/.../pvc-XXX/mount (indicating sdh is the block device).
  
  
• Check Write Latency:
  
  In the SSPI support bundle, review <worker node name>/var/log/stats/iostat.log to check the write latency (w_await) of the PVC's block device. Ideal write latency should be less than 10ms.

Resolution

To permanently resolve this issue, ensure your underlying storage infrastructure meets the required SSP latency specifications.

To recover the system, follow the steps below.

Step 1: Unblock the Database by Expanding PVC Storage

If the Postgres pod is currently crashing, increase the PVC storage to allow Postgres to start up and process its transaction logs. (Note: Your storage class must support volume expansion).

1. SSH to the SSPI appliance using root(SSP 5.0)/sysadmin(SSP 5.1) credentials.
   
   
2. Edit the PVC configuration:
   
   k edit pvc data-postgresql-ha-postgresql-0 -n nsxi-platform

3. Increase the storage request parameter (e.g., change from 50Gi to 100Gi).
   
   
4. Delete the pod to force a restart so it mounts the newly expanded volume:
   
   k delete pod postgresql-ha-postgresql-0 -n nsxi-platform

5. If dependent services (such as the Kafka-broker) are stuck in an Init:CrashLoopBackOff state, delete their pods as well to force a restart.

Step 2: Apply the Remediation Patch

To allow the auto-vacuum process to catch up and prevent future bloat, apply the standalone CronJob patch. This job runs every 30 minutes to mark expired jobs as deleted and clears the payloads of completed and failed jobs, reducing storage consumption.

1. Download the exportjob-cleanup.yaml and apply-exportjob-cleanup.sh scripts attached to this KB.
   
   
2. Copy both files to the SSPI appliance and make the script executable: 
   
   chmod +x apply-exportjob-cleanup.sh

3. Run the script, providing the path to your cluster's kubeconfig.
   
   For example: ./apply-exportjob-cleanup.sh /config/clusterctl/1/workload.kubeconfig

4. To immediately trigger a clean up job, you may run: k create job --from=cronjob/visualization-exportjob-cleanup-cronjob exportjob-cleanup-manual-$(date +%s) -n nsxi-platform

This is tracked internally via the ticket #3706654.

This issue is fixed in SSP version 5.1.1.1.