Generate VCF Support Bundle OnCreateLogBundleFailed BackOffLimitExceeded OOMKilled

Products

• VCF Operations

Issue/Introduction

Unable to generate and download logs successfully from components under VMware Cloud Foundation Operations - Fleet Management - Lifecycle.

When viewing the failed Generate product log bundle Task, the following Error Code and errors are returned:

Request Failed
Generate product log ...
Error Code: OnCreateLogBundleFailed
OnCreateLogBundleFailed
API failed with response: {"statusCode":218,"responseMessage":"status code 218","outputData":"{\"error\":\"job failed: BackoffLimitExceeded; Job has reached the specified backoff limit\"

 

While connected to the VMware Cloud Foundation Automation appliance VM, the corresponding support bundle pods are in OOMKilled state:

1. SSH to the VMware Cloud Foundation Automation appliance VM as vmware-system-user
   
   
2. Enable root and kubectl commands:
   

   sudo su
   export KUBECONFIG=/etc/kubernetes/admin.conf

    

3. Check on the status of the support bundle pods "supportbundle-vmsp-platform-generate-hook":
   

   kubectl get pods -n vmsp-platform | grep supportbundle

   In this scenario, the "supportbundle-vmsp-platform-generate-hook" pods are in failed state with OOMKilled status. Note: These pods quickly clean themselves up after completion or failure.

Environment

VMware Cloud Foundation 9.0.X

Cause

When a Generate product log task is started, VMware Cloud Foundation creates a Kubernetes job and underlying pod.

In this scenario, the underlying pod uses too much memory and it is terminated for exceeding its configured memory limits.

The high memory usage can be caused by generating a large support bundle over a long period of time.

Resolution

Changes must be made to the owning Kubernetes object that generates the "supportbundle-vmsp-platform-generate-hook" job and pods.
Any changes made directly to the job and pod will be reverted or will not take effect.

Reach out to VMware by Broadcom Technical Support referencing this KB article.

Additional Information

Support bundle generation, container size and log collection is improved in VCF 9.1.