Troubleshooting VCF Automation Upgrade Precheck Failures: Staging Binaries Timeout - VCF Automation

Products

• VCF Operations

Issue/Introduction

Upgrade prechecks in VMware Cloud Foundation (VCF) 9.1 may fail during the "Stage VCD_MIGRATOR precheck binaries" task, resulting in a timeout. This article provides a consolidated troubleshooting path for resolving component timeouts caused by pod starvation and fluentd health check failures.

• Precheck task Stage VCD_MIGRATOR precheck binaries times out (approx. 6 hours).
• Logs indicate vmsp_stage_plugin failure.
• System health check VCFMS-HEALTH-002 reports failure.
• Fluentd readiness probe Ready: 0/1.

Environment

• VMware Cloud Foundation 9.1
• VCF Operations 9.1
• VCF Automation 9.1

Cause

1. Pod Starvation (OutOfPods): Excessive accumulation of support-bundle-cluster-info-dump pods exhausts the pod capacity of the node.
2. Fluentd Readiness Probe Failure (VCFMS-HEALTH-002): Log buffer accumulation exceeding 10,000 files causes the logging-operator-fluentd pod to fail health checks, blocking VCF Operations upgrade gates.

Resolution

1. Follow VCF Automation Down Due to OutOfPods Failure If support bundle operator is OOMing. Run remediate-cluster-info-dump.sh to clean stalled jobs.
   
   
2. If VCFMS-HEALTH-002 persists, follow Error: VCFMS-HEALTH-002 - logging-operator-fluentd Readiness Probe Failure in VCF 9.1 To resolve Fluentd Health Checks, purge log buffer chunks causing the readiness failure.
   
   
3. If the issue remains unresolved after remediation, perform a clean redeployment of VCF Automation following: Failed component cleanup during VCF 9.1 installation or upgrade