Bare metal edges from separate NSX-T deployments getting configured with same TEP MAC

Products

• VMware NSX

Issue/Introduction

When multiple Bare Metal Edges are configured on separate NSX-T deployments (managed by different NSX Managers), the same TEP MAC can be allocated to multiple TEPs.

Environment

• VMware NSX-T Data Center
• VMware NSX

Cause

Currently for Bare Metal(BM) Edges, NSX-T uses a fixed range MAC address pool for allocating TEP MAC. When Bare Metal Edges are deployed in different NSX instances, then they may be assigned the same TEP MAC.

Resolution

In order to resolve this issue the following workaround should be applied.

Separate L2 Domains : To avoid issues in TEP traffic drop due to conflicting TEP MAC address on Bare Metal Edges, users need to configure a different "transport_vlan" to separate L2 domains. 

Configure Routing between separate L2 Domains: In order to communicate between different BM Edges with different transport_vlans, users will need to configure routing. For this, user will need to define a new TEP IP pool in different subnet range for BM edges using different NSX deployments. 

Note: Before applying the workaround below, users will need to schedule a maintenance window. This workaround involves down time.

Workaround:

Steps for applying Workaround

Apply below steps for bare metal edges(having same TEP MAC) in different NSX deployments
 

1. Make a copy of the existing BM Edge Hostswitch Uplink profile with new transport_vlan
   1. Use GET https://{manager-ip}/api/v1/host-switch-profiles/<host-switch-profile-id> API to get existing Hostswitch Uplink profile.
   2. Remove below fields from host-switch-profiles API payload which we got in "Step a" and keep other fields in payload as it is.
      1. "id": "xxxx", "display_name": "xxxx", "tags": [], "_create_time": 1709667150680, "_create_user": "admin", "_last_modified_time": 1709873051237, "_last_modified_user": "admin", "_system_owned": false, "_protection": "NOT_PROTECTED", "_revision": 2
   3. Add a meaningful new name with "display_name" field in payload which retrieved in Step 1.2
   4. Change "transport_vlan" in this new host-switch-profiles API payload. This "transport_vlan" should be different from "transport_vlan" of bare metal edge from another NSX deployment.
   5. Finally POST https://{manager-ip}/api/v1/host-switch-profiles API with payload prepared in Step 1.4.
2. Create new TEP IP pool for BM edge node
   Configure a new TEP IP pool for new BM edge, with different subnet network range, as compared to BM edge from another NSX deployment. This new TEP IP pool can be configured from UI as shown below. We also need to mention new TEP Gateway IP for this new TEP IP pool.
   
   
3. Configure external physical switches VLANs and TEP routing
   1. Two separate VLANs must communicate through a layer-3 device, like a router.
   2. Thus, for TEP to TEP communication across these bare metal edges(in different L2 domains i.e. having different VLANs) in different NSX deployment, user needs to configure routing between them externally. 
   3. For bare metal edges from different NSX deployments, TEP IPs should be in different subnet network range. For this configuration we will be using TEP IP pool which we created in Step 3.2.
   4. Make sure TEP gateway IPs on external router are reachable.
   5. Make sure physical switches are configured with correct VLAN.
4. Consume newly created Hostswitch Uplink profile and TEP IP Pool in Bare metal edge node
   Finally consume this newly created Hostswitch Uplink profile(from Step 1) and TEP IP Pool(from Step 2) in Bare metal edge node. For this we can edit bare metal edge node from "System->Fabric→Nodes→Edge Transport Nodes" page and select newly created Hostswitch Uplink profile from drop down as shown in below screenshots.
   
   Also select new newly created TEP ip pool for this BM edge
   
   With this step the Bare Metal Edge will get updated with new TEP IP and new Hostswitch uplink profile. Having different "transport_vlan" makes L2 domain of this BM Edge different as compared to BM Edge from another NSX-deployment. For connectivity between different L2 domains, external TEP to TEP routing needs to be configured by users (as mentioned in Step 3). Thus, having different L2 domains will avoid issues in TEP traffic (such as dropping/flapping) due to conflicting TEP MAC addresses on Bare Metal Edges(across different NSX deployments). 

Additional Information

Impact/Risks:
This will cause connectivity issues in TEP traffic, as multiple Bare Metal Edges are using same TEP MAC address. TEPs tunnels will show as flapping in Edge logs and in packet captures.