VCF Automation Down Due to OutOfPods Failure

Products

• VCF Automation

Issue/Introduction

• The VCF Automation portal is inaccessible via web browser, returning a 500 Internal Server Error.

• New pods cannot be scheduled because the environment has reached the configured pod limit.

• Kubernetes pods in the prelude or vmsp-platform namespaces remain in a Pending or OutOfPods state, can be observed with command

  kubectl -n vmsp-platform get pods

  Note: If kubectl commands fail then please review documentation about enabling SSH access
  
  
• While upgrading VCF Automation, fleet-lcm reports the below error:
  Error Code: LCMVCFA00007
  Error occurred while pushing capabilities in VCF Automation
  com.vmware.vrealize.lcm.vcfa.common.exception.UnableToGetVcfaTokenException: Error occurred while fetching accessToken from refreshToken

Environment

• VCF Automation 9.1.0.0
• VCF Management Services Runtime 9.1.0.0
• Condition: the support-bundle-cluster-info-dump CronJob has been running for a while (it is scheduled hourly), so the defect has had time to accumulate stuck jobs/pods.

Cause

The support-bundle-cluster-info-dump cronjob continuously generates new pods that fail to complete. Over time, these orphaned/stalled pods accumulate until the Kubernetes node reaches its maximum pod capacity (e.g., hitting a limit of 250 pods), preventing functional services from starting or scaling.

Resolution

The issue is resolved with VCF Automation 9.1 Express Patch 2 / VCF Services Runtime 9.1 Express Patch 2

 

Workaround:

This procedure suspends the support-bundle-cluster-info-dump CronJob, fixes its concurrencyPolicy so it stops piling up, cleans up the stalled jobs, then un-suspends. It also cleans up other pods in a stuck state (like seaweedfs-cleanup pods). No VCFA restart is required — freeing the stuck pods lets any starved essential pods reschedule on their own.

 

Important:

This fix disables Flux drift detection so that the changes persist until the next upgrade. An upgrade will re-enable drift detection and also has the permanent fix. Upgrade to 9.1.0.0 EP2 (or later) as soon as possible.

 

Prerequisites

1. Take a backup of VCF Automation / VCF Management Services before proceeding.
2. If the issue was found on VCF Management Services, then enable SSH access

Procedure

1. Download the attached script remediate-cluster-info-dump.sh (see Attachments section below)
2. Copy the script to the VCF Automation / VCF Management Services appliance, replace vsp-appliance-ip with the node IP:

   scp remediate-cluster-info-dump-v2.sh vmware-system-user@<vsp-appliance-ip>:remediate-cluster-info-dump-v2.sh

3.  Connect and elevate to root: 

   ssh vmware-system-user@<vsp-appliance-ip>
   sudo -i

4. Run the script:

   chmod +x /home/vmware-system-user/remediate-cluster-info-dump-v2.sh
   /home/vmware-system-user/remediate-cluster-info-dump-v2.sh

Additional Information

• You can monitor the pod count using kubectl get pods -A | wc -l to ensure you remain below the 250-pod limit.