VCF Automation upgrade from 9.1.0 to 9.1.1 fails due to OOMKilled in walg sidecar container

Products

• VCF Automation

Issue/Introduction

• Upgrading VMware Cloud Foundation (VCF) Automation from version 9.1.0 to 9.1.1 fails during the precheck or submit workflow stage.
• Reviewing the kubectl get pod shows unhealthy pods in the vmsp-platform namespace residing in an Error state:
  
  

  vmsp-platform   upgrade-component-<id>-submit-workflow-with-prechecks-<id>   0/1   Error       0               71m

   

• Checking the PostgreSQL backup console logs (services-logs/prelude/.../backup-container/console-log-*.log) shows an unexpected EOF and exit status 56:
  
  

  stdout F Executing on-demand WAL backup command in target pod ...
  stdout F Master pod identified as vcfapostgres-1 is Running
  stdout F Backup output: curl: (56) OpenSSL SSL_read: OpenSSL/3.5.7: error:0A000126:SSL routines::unexpected eof while reading, errno 0
  stdout F Backup script failed with exit status 56

   

• Inspecting the walg sidecar container status reveals it was terminated by the OOMKiller (Exit Code 137) because it exceeded its 768Mi memory limit:
  
  

  kubectl get pod vcfapostgres-<id> -n prelude -o yaml

  
  

  lastState:
    terminated:
      exitCode: 137
      reason: OOMKilled
  name: walg
  resources:
    limits:
      memory: 768Mi

Environment

VCF Automation 9.1.0

Cause

The default memory limit of 768Mi assigned to the walg sidecar container is insufficient for the tar bundling operations performed during the database backup. When the database size or the volume of logs being processed requires more memory than the limit allows, the Kubernetes Out-Of-Memory (OOM) Killer terminates the container. This causes the backup to abort and the VCF Automation upgrade to fail.

Resolution

Please refer to KB451496 to resolve the issue, and then try upgrading again.

Additional Information

VCF Automation database backup fails with OOMKilled (Exit Code 137) in walg container