VMware Cloud Foundation Cluster expansion fails with insufficient free IPs with blocked IPs

Products

• VMware Cloud Foundation

Issue/Introduction

This will allow users to quickly unblock their cluster expansion operations If they hit a known product issue.

Symptoms:

Cluster expansion of clusters with VVOL as storage type fails with an error stating there are not enough free IPs due to blocked IPs. This is seen in the domainnanager.log at location /var/log/vmware/vcf/domainmanager/domainmanager.log
 

YYYY-MM-YYTHH:MM:SS.079+0000 ERROR [vcf_dm,b08b9173f6773cc4,3bce] [c.v.v.h.s.v.HostNetworkPoolValidator,dm-exec-7]  Insufficient free IPs for VMotion Network. Available IPs 8, Blocked IPs 8, Required IPs 4

Note:

Please note this Knowledge base article is applicable only if:

- Cluster being expanded has VVOL as storage type.
- There are no other running cluster expansion operations.
- There is a sufficient number of available IPs as shown in the logs but there is a number of blocked IPs.
- There were previous cluster expansion operations on that same cluster or another cluster using VVOL as storage type and using ESXi Hosts that are in the same network pool.

Environment

Vmware Cloud Foundation 4.5

Cause

During cluster expansion, when a cluster utilizes vVol as its storage type, the requisite number of IP addresses is reserved from the network pool until these IPs are successfully allocated to the ESXi hosts. Following the allocation of these IPs, the corresponding reserved IPs should be released from the network pool; however, this release is currently not occurring.

Resolution

This issue is fixed with VCF 5.0 release. For VCF 4.x, please proceed with the following workaround:

Note: Please verify there are no running operations on the SDDC Manager. Restarting the domain manager service manually and retrying the cluster expansion operation will remove the stale blocked IPs.

Steps:

1. Open an SSH session to the SDDC Manager VM and switch to the root user using su -.
   
   
2. Restart the domainmanager service:
   
   systemctl restart domainmanager