Expanding VCF 9 cluster fails due to IP Address Pool exhaustion

Products

• VMware NSX
• VMware Cloud Foundation

Issue/Introduction

• Expanding a VCF cluster from SDDC, the cluster cannot be expanded due to a health check failure with the error message:
  "One or more transport node(s) realization failed while creating the transport node collection with profile <uuid> on compute collection <uuid>:domain-### through NSX manager <fqdn>. Remediation: Check the error(s) on NSX Manager <fqdn> and resolve them before restarting the failed task. Reference token: ######."
  
  
  
• In the NSX interface, review System>Fabric>Hosts to observe the Transport Node deployment status. 

1. 1. A cluster shows "Preparation Failed".
      
      
      
      
   2. Within the cluster, it is likely that some hosts show a "Success" status, and some show a "Failed" status. 
      The details for the "Failed" status have the message: "Failed to send the HostConfig message. [TN=TransportNode/<uuid>]. Reason: ipAddressPool path=[/infra/ip-pools/<uuid>] is exhausted, no free IPs available for allocation."
      
      

 

Environment

• VMware Cloud Foundation 9.x
• VMware NSX

Cause

The cluster cannot be expanded because there are not enough IP addresses available for NSX tunnel endpoints.

Resolution

Create a new IP Address Pool with more addresses and update the Transport Node Profile.

1. Identify a larger address space for the tunnel endpoints and Create an IP Pool for Tunnel Endpoint IP Addresses
2. Select the Transport Node Profile (System > Fabric > Profiles > Transport Node Profiles) associated with the cluster, and click Edit.
3. At the IP Pool drop down box, select the newly created IP Pool then chose save.
   
4. Choose the Apply button to complete the updates to the Transport Node Profile.
5. NSX will now update the Transport Nodes with the new profile setting.
6. Re-run the task in SDDC, and the cluster expansion is expected to progress.

Additional Information

Per Changing static pool IP range details post Workload domain deployment, no updates are required to SDDC after making this change in NSX.  VCF does not persist/manage the NSX-T IP pool configuration.