Error: VCFMS-HEALTH-002 VCF Automation Upgrade Failures due to Node IP Pool Exhaustion

Products

• VCF Automation

Issue/Introduction

• VCF upgrade task fails at the precheck or backup stage.
• Log output contains: VCFMS-HEALTH-002.
• Diagnostic error: IP address pool 'unknown-pool' is exhausted (or similar pool exhaustion error).
• Control plane rollout (v1.34.2 to v1.35.6) appears stuck in a loop trying to provision a surge node.

Environment

VCF Automation 9.1.x.

Cause

During a control-plane rolling upgrade, the cluster requires N+1 IP addresses (where N is the number of nodes) to facilitate the creation of a surge node. If the configured node IP pool has exactly N usable IPs—or if usable IPs are incorrectly marked as "excluded"—the surge node fails to provision, the rolling upgrade stalls, and subsequent upgrade tasks fail.

Resolution

Depending on your environment, choose the appropriate remediation path below to ensure sufficient IP availability. Some instances may require both depending on IP range availability.

Prerequisites:

1. SSH into one of the control plane nodes.
2. Set the KUBECONFIG environment variable as per the official VCF documentation.

Option A: Remove Incorrectly Excluded IP Addresses
Use this option if an IP address, such as the bootstrap VM, is incorrectly listed in the exclusion list.

1. Run the command: kubectl edit pd vmsp-platform -n vmsp-platform
2. Navigate down to spec.values.network.ipPool.excludedAddresses.
3. Identify the IP address of the bootstrap VM (or other incorrectly excluded address) and remove it from the list.
4. Save the file by running :wq.
5. Run kubectl get pd vmsp-platform and wait until the status becomes "Successful."

Option B: Expand the Node IP Pool
Use this option if the pool is genuinely exhausted and requires additional IP addresses to accommodate the surge node.

1. Run the command: kubectl edit pd vmsp-platform -n vmsp-platform
2. Navigate to spec.values.network.ipPool.addresses and add the new available IP addresses to the list.
3. Save the file by running :wq.
4. Run kubectl get pd vmsp-platform and wait until the status becomes "Successful."

Verification and Final Steps:

1. Monitor Rollout: Once the configuration is updated, the stuck control-plane rollout will proceed automatically. Monitor the progress using:
   kubectl -n vmsp-platform get machines.cluster.x-k8s.io
   The rollout is complete when all machines are in a Running state and UP-TO-DATE=True.
2. Retry Upgrade: Once the rollout is complete, retry the failed VCF Automation upgrade task within VCF Operations.