Stale Edge Node TEP IP Addresses Not Released After Deletion from vCenter Causing IP Pool Exhaustion

Products

VMware NSX

Issue/Introduction

After deleting an Edge node from vCenter before removing it from NSX-T, the Tunnel Endpoint (TEP) IP addresses remain allocated in the IP pool. This prevents new Edge deployments or reconfigurations due to IP pool exhaustion.

Symptoms include:

Edge node deployment or reconfiguration fails with status pending
TEP IP addresses show as allocated despite Edge node being deleted
Unable to edit or redeploy Edge nodes due to insufficient IP addresses

Steps to validate:

# Check for MP5109 errors in syslog
grep "MP5109" /var/log/syslog*

# Check for MP5109 errors in nsxapi logs
grep "MP5109" /var/log/proton/nsxapi.*

Error messages in logs:

In /var/log/proton/nsxapi.log:

ERROR L2HostConfigTaskExecutor5 IPAMServiceImpl 4433 POOL-MGMT [nsx@6876 comp="nsx-manager" errorCode="MP5109" level="ERROR" subcomp="manager"] Insufficient Resource Exception : no free IP found to allocate from the pool IpPool/<IP-Pool-UUID>

ERROR L2HostConfigTaskExecutor2 IPAMUtils 4433 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP8212" level="ERROR" subcomp="manager"] Failed to get a valid IP from IpPool IpPool/<IP-Pool-UUID> with cidr null, {}.

In /var/log/syslog:

2025-07-15T02:13:29.057Z WARN L2HostConfigTaskExecutor4 IPAMServiceImpl 5140 POOL-MGMT [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Allocation failed for subnet string_id: "<subnet-UUID>" . Retrying from next subnet
2025-07-15T02:13:29.057Z ERROR L2HostConfigTaskExecutor4 IPAMServiceImpl 5140 POOL-MGMT [nsx@6876 comp="nsx-manager" errorCode="MP5109" level="ERROR" subcomp="manager"] Insufficient Resource Exception : no free IP found to allocate from the pool IpPool/<IP-Pool-UUID>.
2025-07-15T02:13:29.057Z ERROR L2HostConfigTaskExecutor4 IPAMUtils 5140 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP8212" level="ERROR" subcomp="manager"] Failed to get a valid IP from IpPool IpPool/<IP-Pool-UUID> with cidr null, {}.
com.vmware.nsx.management.common.exceptions.InsufficientResourcesException: null
    at com.vmware.nsx.management.idas.ipam.service.IPAMServiceImpl.allocate(IPAMServiceImpl.java:105) ~[?:?]
    at com.vmware.nsx.management.switching.sync.host.IPAMUtils.ufoAllocateIpFromPool(IPAMUtils.java:63) ~[?:?]
    at com.vmware.nsx.management.switching.sync.host.VtepPopulator.allocateResourcesToNewVteps(VtepPopulator.java:436) ~[?:?]

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4.x
VMware vSphere 7.x or 8.x

Cause

The Edge node was deleted from vCenter first before proper cleanup in NSX-T. This incorrect deletion sequence causes:

Stale Edge Transport Node entries to remain in the Corfu database
TEP IP addresses to remain allocated in the IP pool
FORCE delete operations do not automatically release TEP IP addresses when the Edge is removed from vCenter before NSX-T cleanup

Resolution

Step 1: Clean Up Stale Edge Node Entries

Remove stale Edge entries from the NSX Manager using the API available from NSX-T 3.2.1:

curl -u admin -k -X POST https://127.0.0.1/api/v1/transport-nodes?action=clean_stale_entries

Reference: Cleanup Stale Edge Nodes present in the NSX-T UI from the Corfu database

Step 2: Identify IP Pool and Allocated IPs

Get the IP pool UUID:

curl -u admin -k -X GET https://127.0.0.1/api/v1/pools/ip-pools/

Check current IP allocations in the pool:

curl -u admin -k -X GET https://127.0.0.1/api/v1/pools/ip-pools/<IP-Pool-UUID>/allocations

Step 3: Release Stale IP Addresses

Release each stale IP address from the pool:

curl -u admin -k -X POST "https://127.0.0.1/api/v1/pools/ip-pools/<IP-Pool-UUID>?action=RELEASE" \
  -H "Content-Type: application/json" \
  -d '{"allocation_id": "<IP-ADDRESS>"}'

Replace <IP-Pool-UUID> with the actual pool ID and <IP-ADDRESS> with each stale IP that needs to be released.

Step 4: Verify IP Release

Confirm the IP addresses have been released:

curl -u admin -k -X GET https://127.0.0.1/api/v1/pools/ip-pools/<IP-Pool-UUID>/allocations

Step 5: Retry Edge Configuration

After releasing the IP addresses, the Edge node configuration should complete successfully. Navigate to the Edge node in the NSX-T UI and click Edit to reconfigure.

Important: If the error persists after following these steps, contact Broadcom Support for further assistance.