NSX-T Federation Edges are getting RTEPs IPs from stale IP pool

Products

VMware NSX

Issue/Introduction

You are using a federation setup.
RTEP's are configured and using respective IP pools.
You have decided to change the IP ranges for the RTEP's.
To do this, you have deleted the old RTEP pool and added a new RTEP pool.
On checking the edge nodes for the respective site, we see the RTEP IP did not change and are still using the old pool addresses.
- SSH to the edge node as admin user
- Run 'get logical-router' and identify the vrf for 'RTEP_TUNNEL '
- Then change into that VRF using the command 'vrf <vrf-id>'
- Now to find the IP address assigned to the RTEP interface run 'get interface' and look for the interface with name 'remote-tunnel-endpoint' the IP address is listed there:
```
Interface     : <interface-uuid>
    Ifuid         : 283
    Name          : remote-tunnel-endpoint
    Fwd-mode      : IPV4_AND_IPV6
    Internal name : uplink-283
    Mode          : lif
    Port-type     : uplink
    IP/Mask       : 192.168.1.10/24;<ip-6-address>(A)
    MAC           : <MAC-ADDRESS>
```

On the local manger site log /var/log/proton/nsxapi.log, there may have an error similar to:

2023-05-30T12:46:59.864Z ERROR ActivityWorkerPool-1-11 IPAMServiceImpl 4872 POOL-MGMT [nsx@6876 comp="nsx-manager" errorCode="MP600" level="ERROR" subcomp="manager"] IP Pool not found for id IpPool/<Pool-UUID>
2023-05-30T12:46:59.865Z WARN ActivityWorkerPool-1-11 ActivityExecutor 4872 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Phase execution failed. Activity= Activity= 'RtepConfigUpdateActivity:<UUID>' entity= 'EdgeTransportNode/<Edge-UUID>', Error= 'ActivityFailureException', phase= '{}' >>>>>>> failure on RTEP config update activity for Edge Node 2 "<Node-Name>"
com.vmware.nsx.management.common.exceptions.ObjectNotFoundException: null
        at com.vmware.nsx.management.idas.ipam.service.IPAMServiceImpl.getPoolIfExists(IPAMServiceImpl.java:918) ~[?:?]
        at com.vmware.nsx.management.idas.ipam.service.IPAMServiceImpl.release(IPAMServiceImpl.java:612) ~[?:?]
2023-05-30T12:48:27.376Z WARN ActivityWorkerPool-1-0 ActivityExecutor 4872 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] Phase execution failed. Activity= Activity= 'RtepConfigUpdateActivity:<UUID>' entity= 'EdgeTransportNode/<Edge-UUID>', Error= 'ActivityFailureException', phase= '{}'
624097: at com.vmware.nsx.management.edge.federation.RtepConfigUpdateActivity.phaseBegin(RtepConfigUpdateActivity.java:43) ~[?:?] >>>>>>> failure on RTEP config update activity for Edge Node 1 "<Node-Name>"

After replacing an old RTEP pool with new one, Edges are still getting IPs from stale RTEP pool

Environment

VMware NSX-T Data Center 3.x

Cause

The edge didn't receive correct RTEP IP information from controller due to stale entry in the Corfu database.

Resolution

If the current IP pool has IP addresses allocated, it is not possible to edit the pool.

From VMware NSX-T Data Center 3.2.3 onwards, an alert will be generated to warn that some IP addresses are still allocated and prevent IP pool modification.

This issue is resolved in VMware NSX-T Data Center 3.2.3, available by following instructions found at Downloading Broadcom products and software.

Workaround

If you believe you have encountered this issue and are unable to upgrade, please open a support request with Broadcom Support and refer to this KB article.
For more information, see How to Submit a Support Request.

▶