VCF 9.1 Multi-Component Upgrade Stalls with VCFMS-HEALTH-002 Health Check Error

Products

• VCF Operations

Issue/Introduction

• During a VCF precheck or upgrade task, the Platform Health Check fails with error code [VCFMS-HEALTH-002]. The error reflects a stall in the VMSP PackageDeployment operator. In VCF 9.1, this is frequently caused by a resource state conflict in the logging-operator-fluentd component. When log buffer files exceed the default threshold (10,000), the pod fails readiness checks, causing the VMSP platform to suspend all active package upgrades to prevent further inconsistency.

• The UI displays the following error message under Platform Health Check:

  Platform Health Check Error [platform-daemonsets-core : 1 of 13 resources are not ok: logging-operator-fluentbit: wrong resource state: InProgress - Available: 0/1;] [VCFMS-HEALTH-002]

• Prechecks remain stuck at the "Binary Staging" stage indefinitely.
  
  
• VMSP tasks for the affected components show a status of Pending or IN_PROGRESS without resolution.
  
  
• Argo workflows are not created or reconciled for the affected tasks.

Environment

• VMware Cloud Foundation 9.1
• Multi-component patching (VCF Operations, Automation, Identity Broker)
• VMware Management Services Platform (VMSP)

Cause

These failures often stem from cluster resource constraints, specifically an OutOfPods error state triggered by support-bundle-cluster-info-dump processes. When the cluster enters this state, the VMSP service cannot successfully reconcile or spawn the necessary Argo workflows to execute the staging jobs.

Resolution

To resolve this issue, please follow these steps:

Step 1: Investigate Cluster Health for OutOfPods Errors

1. Check the health of the VCFA or management cluster to verify if pods are failing due to OutOfPods conditions.

2. Refer to KB VCF Automation Down Due to OutOfPods Failure for detailed diagnostic and remediation guidelines regarding OutOfPods error states.

Step 2: Manually Clean Up Affected Pods

1. Identify all pods currently stuck in an OutOfPods state:
   

   kubectl get pods -A | grep OutOfPods

2. Manually delete the stuck pods to allow Kubernetes to recreate them:
   

   kubectl delete pod <pod-name> -n <namespace>

3. Monitor the newly generated pods and confirm they successfully transition to a Running state.

Step 3: Retry Prechecks

1. Once all cluster resources stabilize and pods report healthy states, log in to the VCF UI.

2. Re-run the precheck task.

Step 4: Perform Component-Specific Cleanup if prechecks continue to hang after cluster health is restored, the VMSP task state may be corrupted:

1. Execute the cleanup_component.py script to clear stale task states for the specific failing components (e.g., VCFA, VCD_MIGRATOR).

2. Trigger a fresh precheck from the VCF UI.

Step 5: To clear the health check error and allow patching to resume, increase the Fluentd buffer threshold and clear the stale buffer files as outlined in KB: Error: VCFMS-HEALTH-002 - logging-operator-fluentd Readiness Probe Failure in VCF 9.1