NSX host preparation fails with error: Insufficient IP addresses in IP pool

Products

• VMware NSX

Issue/Introduction

Symptoms:

• IP addresses assigned from the configured VXLAN Tunnel End Point (VTEP) pool to a prepared NSX for vSphere 6.x host are not shown as allocated by NSX
• New ESXi hosts added to the same cluster are assigned to the same allocated IP addresses, creating duplicate IP address assignments
  
• NSX host preparation fails for some hosts
• You see the error:
  
  Insufficient IP addresses in IP pool.
  

Environment

VMware NSX for vSphere 6.0.x
VMware NSX for vSphere 6.2.x
VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.1.x

Cause

This issue occurs because all of the IP addresses assigned to a pool have been assigned.

Note: This condition may be expected or unexpected.

A duplicate IP assignment issue occurs because of an IP address conflict between VTEPs. IP addresses are released automatically when they are not in use or when a host is unconfigured and the VTEPs are removed. However, IP addresses may continue to be assigned and can result in duplicate assignments under one of these conditions:

• A prepared host is removed directly from vCenter Server.
• A cluster is removed directly from vCenter Server without doing the unconfigure action on NSX.

This condition also has been seen as a side-effect of a configuration error between the teaming policy and the required number of VTEPs on a prepared host. For example, a teaming policy of source port ID hashing with four active uplinks requires four VTEPs and an IP pool of four addresses. If the teaming policy is changed directly in vCenter Server which is an unsupported operation with NSX, the NSX VXLAN module creates the VTEPs based on its understanding of the teaming policy. Specifically, it may create only one VTEP and report that additional VTEPs could not be created due to IP address exhaustion.

Note: Creating VTEPs in vCenter Server or changing the teaming policy directly in vCenter Server is unsupported with NSX. The NSX Manager maintains a database of the VTEPs created automatically on each ESXi host. Creating VTEPs manually in vCenter Server bypasses this database process, resulting in unpredictable system behavior, including no accidental deletion checks or associated alarms, no display of the VTEPs in the NSX User Interface (UI), and no automatic clean-up when a host or cluster is unprepared for VXLAN.

This issue may also occur in this example:

User has 3 uplinks per host, 2 hosts, so IP pool needs 6 IP addresses. Previous to this attempt, the user corrected the number of Uplinks in the DVS using vCenter, to 2 per host, so only allocated 5 IP addresses - 4 + 1 spare. But the NSX Manager has no knowledge of this, so goes off the prior config of 3 VTEPs x 2 = 6 IP addresses.

In this scenario:

1. Increase the number of IP address to 6, and then attempt the configuration again. In the vsm.log, you see entries similar to:
   
   VXLAN_VMKNIC_PG_CREATION_FAILED.
   
   
2. Unconfigure VXLAN.
3. Configure VXLAN and this time the correct number of uplinks (2 in this example) will be shown and the configuration should be expected to succeed without error.
   

Resolution

This is a known issue affecting VMware NSX for vSphere 6.x.

Currently, there is no resolution.

To work around the issue, the following API call can be used to release the IP addresses that are not in use.

Note: In NSX for vSphere release 6.2.0 and later, if a VTEP IP address is changed directly on a host or in vCenter Server, the old IP address of the VTEP is released automatically.

To determine the IP addresses:

1. Log in to the ESXi host as root using SSH or direct console.
2. Run the esxcfg-vmknic -l command.
3. POST https://<nsxmgr-ip>/api/2.0/services/ipam/pools/ipaddresspool-2/ipaddresses
   Body:
   <ipAddressRequest>
   <allocationMode>RESERVE</allocationMode>
   <ipAddress>xxx.xxx.xxx.xxx</ipAddress>
   </ipAddressRequest>

 

Additional Information