GM created Stale DHCP Relay Profiles observed on LM under Manager UI

Products

VMware NSX

Issue/Introduction

DHCP Relay Profile created on Global Manager.
That DHCP Relay Profile is being called under segments which are being used in Local sites.
On Local Manager under Policy UI, we are able to see the respective entry for DHCP Relay Profile.
On Local Manager under Manager UI > Networking > DHCP > Relay Profiles and Relay Services, we would see multiple profiles are being created for the same segment + DHCP Relay.

Relay Profile:

Relay Services:
In cases, if you update DHCP server IP address in the DHCP Relay profile (From GM), on LM Manager UI > Networking > DHCP > Relay Profiles and Relay Services, we would be seeing new profile entries are being created using the updated IP address of the DHCP server. However the old Relay profiles and Relay Services pointing towards the old DHCP server IP remains present on the Manager UI/API.

DHCP server IP address updated on the DHCP Relay profile on GM:

New DHCP Profile and Services entries seen in LM Manager UI pointing to this new IP:

However the old profiles and services for the old DHCP server IP still remains in Manager API/UI:

Usually, even though these old Relay Profiles and Relay Services entries remains present on Manager UI/API, the updated one only gets pushed on the dataplane towards the Transport nodes. To validate which DHCP Server IP is being associated on a segment on dataplane, you can login to a Host Transport Node hosting a VM connected on the respective segment and check the following:

> get logical-switches

                     Overlay LCP Entry
============================================================
  VNI              Logical Switch UUID              Name
 72705     8d0a0e98-####-####-####-d7977d8c6e43   TestSegment-01_SiteA


> get logical-routers

                                              Logical Routers Summary
------------------------------------------------------------------------------------------------------------------
               VDR UUID                LIF num   IPv4 Route num   IPv6 Route num  Max Neighbors  Current Neighbors
 6300aa55-####-####-####-78c355646f16     2             3                5            50000              8


> get logical-router 6300aa55-####-####-####-78c355646f16 interfaces

                         Logical Router Interfaces
---------------------------------------------------------------------------
IPv6 DAD Status Legend:  [A: DAD_Sucess], [F: DAD_Duplicate], [T: DAD_Tentative], [U: DAD_Unavailable]

LIF UUID                 : 39cfe27c-####-####-####-f7fbd3285093
Mode                     : [b'Routing']
Overlay VNI              : 72705
IP/Mask                  : 172.16.110.1/24
Mac                      : 02:50:56:##:##:##
Connected DVS            : RegionA01-VDS7
Control plane enable     : True
Replication Mode         : 0.0.0.1
Multicast Routing        : [b'Enabled', b'Oper Down']
State                    : [b'Enabled']
Flags                    : 0x84388
DHCP relay               : 192.168.120.151 >>>
DAD-mode                 : ['LOOSE']
RA-mode                  : ['UNKNOWN']

Environment

VMware NSX-T Data Center
VMware NSX

Cause

This is a corner case issue with stale DHCP relay profiles due to some race conditions.

Resolution

Workaround:

1. In most of the cases the multiple entries of DHCP Relay profiles and services are not going to cause any problem on dataplane, as long as the correct profile with the intended DHCP server is pushed towards the transport nodes. This we can verify using the method described on #6 under issue/introduction section.

2. However, in case we see that for a specific segment, the updated DHCP server IP is not pushed towards the Transport nodes, we can create a new DHCP Relay profile using the updated DHCP server IP and call it under that segment. This will push the new DHCP Relay profile for that segment towards the Transport nodes.