VCF Automation Down Due to OutOfPods Failure

Products

• VCF Automation

Issue/Introduction

• The VCF Automation portal is inaccessible via web browser, returning a 500 Internal Server Error.

• New pods cannot be scheduled because the environment has reached the configured pod limit.

• Kubernetes pods in the prelude or vmsp-platform namespaces remain in a Pending or OutOfPods state, can be observed with command

  kubectl -n vmsp-platform get pods

  Note: If kubectl commands fail then please review documentation about enabling SSH access
  
  
• While upgrading VCF Automation, fleet-lcm reports the below error:
  Error Code: LCMVCFA00007
  Error occurred while pushing capabilities in VCF Automation
  com.vmware.vrealize.lcm.vcfa.common.exception.UnableToGetVcfaTokenException: Error occurred while fetching accessToken from refreshToken

Environment

• VCF Automation 9.1.0.0
• VCF Management Services Runtime 9.1.0.0

Cause

The support-bundle-cluster-info-dump cronjob continuously generates new pods that fail to complete. Over time, these orphaned/stalled pods accumulate until the Kubernetes node reaches its maximum pod capacity (e.g., hitting a limit of 250 pods), preventing functional services from starting or scaling.

Resolution

The issue is resolved with VCF Automation 9.1 Express Patch 2 / VCF Services Runtime 9.1 Express Patch 2

 

Workaround:

This procedure suspends the support-bundle-cluster-info-dump CronJob, fixes its concurrencyPolicy so it stops piling up, cleans up the stalled jobs, then un-suspends. No VCFA restart is required — freeing the stuck pods lets any starved essential pods reschedule on their own.

Important:

Flux drift detection is left enabled, so the concurrencyPolicy patch this script applies will eventually be reverted on its own. Upgrade to 9.1.0 EP2 (or later) as soon as possible. If the upgrade must be delayed, re-run this script periodically in the meantime to keep the CronJob from piling up again.

Prerequisites

1. Take a backup of VCF Automation / VCF Management Services before proceeding.
2. If the issue was found on VCF Management Services, then enable SSH access

Procedure

1. Download the attached script remediate-cluster-info-dump.sh (see Attachments section below)
2. Copy the script to the VCF Automation / VCF Management Services appliance, replace vsp-appliance-ip with the node IP:

   scp remediate-cluster-info-dump.sh vmware-system-user@<vsp-appliance-ip>:remediate-cluster-info-dump.sh

3.  Connect and elevate to root: 

   ssh vmware-system-user@<vsp-appliance-ip>
   sudo -i

4. Run the script:

   chmod +x /home/vmware-system-user/remediate-cluster-info-dump.sh
   /home/vmware-system-user/remediate-cluster-info-dump.sh

Additional Information

• You can monitor the pod count using kubectl get pods -A | wc -l to ensure you remain below the 250-pod limit.