VMware Runtime Services upgrade Fails due to IP Address Pool Exhaustion

Products

• VCF Operations

Issue/Introduction

When upgrading VMware Cloud Foundation (VCF) from version 9.1 to 9.1.1, the upgrade workflow fails during the VCF Runtime Services component upgrade step.

In the management user interface, the following task failure and error messages are displayed:

Step vmsp_upgrade failed. Reference Code: ########. Check logs for details

IP address pool 'unknown-pool' is exhausted. N/A. No available IP addresses for new nodes. Expand the IP pool for the VCF services runtime using the "Expand Node IP Pool" action menu.

The management components remain partially upgraded, and the upgrade cannot proceed until additional IP addresses are allocated.

Environment

• VMware Cloud Foundation 9.1.x

Cause

During the VCF Runtime Services upgrade from 9.1 to 9.1.1, new service nodes and containers are provisioned concurrently before legacy containers and nodes are decommissioned. If container backup timeouts or retries occur during vmsp_upgrade, the temporary IP consumption can exceed the initial static allocation (even if initially provisioned to the recommended maximum of 30 IP addresses). Once the assigned pool has no free leases remaining, node provisioning halts and reports IP pool exhaustion.

Resolution

To resolve this issue, expand the IP pool allocated to the VCF Runtime Services component and resume the upgrade:

1. Log in to the VMware Cloud Foundation management interface.
2. Navigate to the VCF Services Runtime management section.
3. Open the Actions dropdown menu and select Expand Node IP Pool.
4. Allocate additional IP addresses to the IP pool (for example, add an additional range of available IP addresses to accommodate temporary upgrade concurrency).
5. Confirm and save the expanded IP pool configuration.
6. Return to the upgrade task interface and initiate a retry of the vmsp_upgrade step.
7. Monitor the task until the VCF Runtime Services upgrade completes successfully.