VCF Automation database backup fails with OOMKilled (Exit Code 137) in walg container

Products

• VCF Operations/Automation (formerly VMware Aria Suite)

Issue/Introduction

Database backups for VCF Automation (VCFA) fail consistently during the backup process. Logs indicate that the backup script fails with exit code -9 (SIGKILL). Upon inspecting the pod status, the walg sidecar container is found to be in an OOMKilled state (Exit Code 137), indicating that the process is exceeding its allocated memory limit.

Symptoms:

• The walg container logs show: reason: OOMKilled and exitCode: 137.
• The kubectl describe pod output for the PostgreSQL instance confirms that the walg container limit is set to 768M

  limits: cpu: "1" memory: 768Mi

• Backup failures occur consistently during the tar bundling phase ("Starting part 2"), where memory consumption spikes.

Environment

VCF Automation 9.x

Cause

The default memory limit of 768Mi assigned to the walg sidecar container is insufficient for the tar bundling operations performed during the database backup, specifically when the database size or the volume of logs being processed leads to memory exhaustion. This results in the OOMKiller terminating the container.

Resolution

To resolve this, you must increase the postgresBackupSidecarMemoryLimit in the PackageDeployment for vmsp-platform.

1. Edit the PackageDeployment: Execute the following command to edit the vmsp-platform deployment:

   kc edit pd vmsp-platform
   

2.  Update the memory limit: Locate the postgresBackupSidecarMemoryLimit parameter and update it from 768Mi (or current value) to 1Gi:

   values: postgresBackupSidecarMemoryLimit: 1Gi

    
   Apply and Monitor: Save and exit the editor. The deployment will automatically apply the changes and restart the affected pods. Once the pods are stable and running with the new limit, retry the database backup from the VCF UI.