NSX-T Edge tunnels are down to ESXi hosts when sharing the same VLAN for TEP traffic

Products

• VMware NSX

Issue/Introduction

• NSX-T datacenter 3.x and NSX 4.x
• Hosts are configured to use NVDS or VDS host switch.
• NSX-T Edge Nodes vNICs are connected to NSX-T segments.
• BFD sessions between Hosts and Edge Nodes are down.
• BFD sessions between Edges Nodes are up.
• BFD sessions between Hosts are up.
• Hosts and Edge Nodes are configured to use the same VLAN for TEP traffic.
• Edge TN has been created before the Host TN it resides on: creation time for Transport Nodes can be verified by running the below API call (In the given example, it can been seen that creation time of the Edge node is before the creation time of the TN it reside on) :

GET https://{nsx manager IP}/api/v1/transport-nodes

"_create_time": 1674629396853,
"_create_user": "system",
"_last_modified_time": 1674629396853,
"_last_modified_user": "system",
"_protection": "NOT_PROTECTED",
"_revision": 0,
"_system_owned": false,
"description": "",
"display_name": "esx01",

"_create_time": 1673985627749,
"_create_user": "admin",
"_last_modified_time": 1674628788051,
"_last_modified_user": "admin",
"_protection": "NOT_PROTECTED",
"_revision": 8
"_system_owned": false,
"display_name": "edge01",
 

Note: The times above are epoch times. The human readable times for above are below:

1674629396853 = Wednesday, January 25, 2023 6:49:56.853 AM
1673985627749 = Tuesday, January 17, 2023 8:00:27.749 PM

Environment

VMware NSX 

VMware NSX-T Datacenter

Cause

• If an Edge Node is deployed on an NSX-T prepared Host and both the Edge Node and Host use the same IP subnet/VLAN for the TEP interface, there is a port property called nestedTNConfig sent by the Management Plane (MP) to the host when the Edge TEP vNIC is attached to a NSX port group / Segment.
• This issue occurs when an Edge Node was deployed on a Host that was not prepared with NSX, and the nestedTNConfig property is not sent.
• Later the Host is prepared for NSX with an NVDS or VDS host switch and this causes hosts only listens to traffic for its TEP interface and drop all traffic for the TEP interface of the Edge Node.

Resolution

Workaround:

There are two workarounds for this issue:

1. Put the TEP traffic of the Edge Node on a different VLAN than the Host’s TEP VLAN.

• Create a new uplink profile with a new VLAN. If IP pool is used, create a new IP pool, which allocates IP addresses in the newly created VLAN.
• Apply this profile to the Edge Node.

Note: To implement this Workaround, a separate subnet/VLAN will be needed.
 

2. Create a new NVDS/VDS for the ESXi Host and assign a different uplink (vmnic) to the NVDS/VDS.  Apply the same uplink profile (including VLAN) and IP pool to the new uplink.
 

Note: To implement this workaround, additional vmnic(s) will be required for the new NVDS/VDS