Upgrading VCF Automation failed due prechecks

Products

• VCF Operations

Issue/Introduction

• In VCF Ops 9.1, you observe the following errors:

- > Build > Lifecycle > VCF Management > Update > Run precheck > 

Migration service engine
##-###  9.1.0.0253  to 9.1.0.0255 > Ready for upgrade pre checks failed  >

Subtask 
-Set VCD_MIGRATOR upgrade context for the workflow // passed 
-Stage VCD_MIGRATOR precheck binaries ---> Failed --> Error: Step stage_vsp_cluster_plugin failed. Reference Code: ########. Check logs for details.

- > Build > Lifecycle > VCF Management > Update > Run precheck > 
VCF Automation
##-###  9.1.0.0254  to 9.1.0.0255 > Ready for upgrade pre checks failed >

Subtask
-Fetch Source Component Details  // passed 
-Initialized Required Parameters for the Upgrade  //passed 
-Stage VCF Automation precheck binaries  ---> Failed --> Error:  Step stage_vsp_cluster_plugin failed. Reference Code: ########. Check logs for details.

• Kubernetes pods in the prelude or vmsp-platform namespaces remain in a Pending or OutOfPods state. You can observe this with the following command:

  kubectl -n vmsp-platform get pods

  Note: If kubectl commands fail, please review the documentation about enabling SSH access.

Environment

VCF Operations 9.1.0

VCF Automation 9.1.0 

Cause

This issue occurs due to pod starvation in a single-node VCF Automation (VCFA) deployment. An excessive accumulation of cluster-info-dump pods (> 300) exhausts the available pod capacity. Consequently, Kubernetes (K8s) is unable to schedule new pods, resulting in an OutOfPods error that prevents workflows from running.

Resolution

1. Take a backup of VCF Automation / VCF Management Services before proceeding.
2. Follow the steps in this KB article: VCF Automation Down Due to OutOfPods Failure.