VMware Cloud Foundation Cluster expansion fails with insufficient free IPs with blocked IPs

Products

• VMware Cloud Foundation

Issue/Introduction

This will allow users to quickly unblock their cluster expansion operations If they hit a known product issue.

Symptoms:
Cluster expansion of clusters with VVOL as storage type fails with an error stating there are not enough free IPs due to blocked IPs. This is seen in the logs like below:
 

2022-12-01T15:18:33.079+0000 ERROR [vcf_dm,b08b9173f6773cc4,3bce] [c.v.v.h.s.v.HostNetworkPoolValidator,dm-exec-7]  Insufficient free IPs for VMotion Network. Available IPs 8, Blocked IPs 8, Required IPs 4

Note:
Please note this Knowledge base article is applicable only if:

- Cluster being expanded has VVOL as storage type.
- There are no other running cluster expansion operations.
- There is a sufficient number of available IPs as shown in the logs but there is a number of blocked IPs.
- There were previous cluster expansion operations on that same cluster or another cluster using VVOL as storage type and using ESXi Hosts that are in the same network pool.

Environment

VMware Cloud Foundation 5.0
Vmware Cloud Foundation 4.5

Cause

During cluster expansion, when the cluster has VVOL as storage type, the number of required IPs are being blocked (reserved) from the network pool until they are allocated to the ESXi Hosts. Once IPs are allocated, the number of blocked (reserved) required IPs must be released but they are not. 

Resolution

This issue will be fixed with VCF 5.0 release.

Workaround:
Note: 
Please verify there are no running operations on the SDDC Manager.
 

Restart the domain manager service manually and retry the cluster expansion operation. Restarting the domain manager service will remove the stale blocked IPs.

 

1. Open a SSH session to the SSDC Manager VM and change to the root user su -
2. Restart the domain manager service
systemctl restart domainmanager