VCF Automation Upgrade to 9.1 Fails with "VCFMS-HEALTH-002", IP Collisions, and "no healthy upstream"

Products

• VCF Automation

Issue/Introduction

During an attempt to upgrade VMware Cloud Foundation (VCF) Automation from version 9.0.2 to 9.1 via Fleet Management, the upgrade prechecks fail due to multiple platform health check errors. Concurrently, the VCF Automation portal becomes inaccessible, displaying a blank screen with a "no healthy upstream" message.

Specific Errors Observed:

• Platform Health Check Error [VCFMS-HEALTH-002] across multiple resources (deployments, machines, control plane).

• One or more IP addresses in the configured node pool are already in use on the network.

• UI Error: no healthy upstream

Backend Kubernetes investigation reveals that multiple core resources are stuck in an InProgress or failing state, and the cluster is attempting to reference a vSphere machine template that no longer exists.

Environment

 

• Product: VCF Automation (VCFA)

• Management Component: VCF Operations (Fleet Management)

• Version: Upgrading from 9.0.2 to 9.1

 

Cause

There are two sequential causes blocking the upgrade process:

1. Missing Machine Template: The system is attempting to reference a 9.0.2 base vSphere template located on a decommissioned datastore. Without this valid template, Kubernetes cannot provision or stabilize the cluster nodes, breaking the environment and causing the "no healthy upstream" and VCFMS-HEALTH-002 errors.

2. IP Address Collisions: After addressing the missing template, the upgrade precheck actively scans the network pool. Because there are stale IP address claims and orphaned Kubernetes resources with stuck finalizers holding onto the target node pool IPs, the system blocks the upgrade to prevent duplicate IP assignments.

Resolution

To stabilize the 9.0.2 cluster and successfully complete the 9.1 upgrade, you must manually correct the missing template reference and clear the IP collisions.

Phase 1: Fix the Missing Template

Restoring the correct 9.0.2 vSphere template and patching the control plane allows Kubernetes to successfully provision the nodes.

1. Restore the Base Template: Download the vmsp-vcfa-combined-9.0.2 package, import the correct vcf-services-platform-template into vCenter, and convert it to a template.

2. Create Kubernetes Reference: In the Kubernetes backend, create a new VSphereMachineTemplate pointing to the correct, updated vCenter path.

3. Patch the Control Plane: Update the KubeadmControlPlane to use the new template by running the following patch command:

   Bash

    

   kubectl patch kcp <kcp-name> --type=merge -p '{"spec":{"machineTemplate":{"infrastructureRef":{"name":"<new-template-name>"}}}}'
   

4. Trigger Recreation: Delete the failed machine pods to trigger a recreation. This initiates a rolling update to stabilize the 9.0.2 nodes and resolves the "no healthy upstream" error.

Phase 2: Resolve IP Collisions

Clearing the stale IP claims removes the network collisions blocking the precheck.

1. Delete Stale Claims: Clean up stale IP address claims that are blocking allocation by executing:

   Bash

    

   kubectl delete ipaddressclaim <claim-name> -n vmsp-platform
   

2. Remove Stuck Finalizers: If resources remain stuck in a terminating state, remove their finalizers (e.g., using kubectl replace with jq edits) to force Kubernetes to delete the orphaned objects.

3. Reconfigure Network (If Applicable): If required by your topology, reconfigure the upgrade to use a new IP block and new FQDN as per Broadcom KB 443199.

4. Retry Precheck: Retry the 9.1 upgrade precheck in Fleet Management. With a healthy baseline and available IP addresses, the precheck will pass and the upgrade can proceed.