Fix scale out issue due to insufficient IP addresses available in the VCF Services Runtime IP pool

Products

• VMware Cloud Foundation
• VCF Operations

Issue/Introduction

• On the VCF Installer or LCM UI, during the Install/Upgrade, the Scale-out operation process appears to hang indefinitely.
• No immediate error is displayed while the system attempts to retry the task (up to 3 times).
• Timeout & Final Error: A user-facing error is only returned after the workflow reaches its timeout (default 6 hours). At that point, the failed task will display the following message:

"IP address pool 'management-global-ip-pool' is exhausted. Total: X, Used: X, Free: 0. No available IP addresses for new nodes. Expand the IP pool for the VCF services runtime using the 'Expand Node IP Pool' action menu."

Error ID: com.vmware.vcfms.infrastructure.ipam.PoolExhausted

Environment

VCF Services Runtime 9.1

Cause

Provided IP pool does not have enough number of IPs to allocate to more nodes.

 

Resolution

1. Open VCF Operations UI and navigate to Build > Lifecycle > Components tab
2. Click on the VCF Services Runtime link.
3. Scroll down the page to the Nodes section.
4. Identify a Control Plane node and take note of the IP address.
5. SSH to this Control Plane node as vmware-system-user and provide the password for the account.
6. Elevate to root (re-enter vmware-system-user the password).
   1. sudo -i
7. Execute the attached script
   chmod +x ip-pool-update.sh
./ip-pool-update.sh <ipv4_list> [ipv6_list]
   
   This is a replace rather than a merge, so the request needs to contain the complete list of IPs.

Script usage:

Usage: ../../../../vmsp/scripts/manual-kb/ip-pool-update.sh <ipv4_list> [ipv6_list]

Arguments:
  ipv4_list            Comma-separated IPs, ranges, and/or CIDRs for ipPool.
  ipv6_list            Optional; same for ipPoolv6. Omit to leave ipPoolv6 unchanged.

Nodes will be provisioned and all machines will be in running phase

kubectl get machine -A
NAMESPACE       NAME                                   CLUSTER      NODE NAME                              READY   AVAILABLE   UP-TO-DATE   PHASE     AGE     VERSION
vmsp-platform   management-default-{UNIQUE_ID}   management   management-default-{UNIQUE_ID}   True    True        True         Running   3h57m   v1.34.2
vmsp-platform   management-default-{UNIQUE_ID}   management   management-default-{UNIQUE_ID}   True    True        True         Running   14m     v1.34.2
vmsp-platform   management-default-{UNIQUE_ID}   management   management-default-{UNIQUE_ID}   True    True        True         Running   3h57m   v1.34.2
vmsp-platform   management-default-{UNIQUE_ID}   management   management-default-{UNIQUE_ID}   True    True        True         Running   3h57m   v1.34.2
vmsp-platform   management-{UNIQUE_ID}           management   management-{UNIQUE_ID}           True    True        True         Running   3h57m   v1.34.2