NSX-T Edge tunnels are down to ESXi hosts when sharing the same VLAN for TEP traffic

Products

• VMware NSX

Issue/Introduction

Symptoms:

• You are running one of the NSX-T versions from 3.2 or 4.0.
• Hosts are configured to use NVDS or VDS host switch.
• NSX-T Edge Nodes vNICs are connected to NSX-T segments.
• BFD sessions between Hosts and Edges Nodes are down.
• BFD sessions between Edges Nodes are up.
• BFD sessions between Hosts are up.
• The Host and Edges Nodes are configured to use the same VLAN for TEP traffic.
• When you run the following API call you should expect to see the create time for Transport Nodes (the creation time of the Edge Nodes are before the transport nodes they reside on) :

GET https://{nsx manager IP}/api/v1/transport-nodes

"_create_time": 1674629396853,
"_create_user": "system",
"_last_modified_time": 1674629396853,
"_last_modified_user": "system",
"_protection": "NOT_PROTECTED",
"_revision": 0,
"_system_owned": false,
"description": "",
"display_name": "esx01",

"_create_time": 1673985627749,
"_create_user": "admin",
"_last_modified_time": 1674628788051,
"_last_modified_user": "admin",
"_protection": "NOT_PROTECTED",
"_revision": 8
"_system_owned": false,
"display_name": "edge01",
 

Note: The times above are epoch times. The human readable times for above are below:

1674629396853 = Wednesday, January 25, 2023 6:49:56.853 AM
1673985627749 = Tuesday, January 17, 2023 8:00:27.749 PM

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center 4.x
VMware NSX-T Data Center

Cause

• If an Edge Node is deployed on an NSX-T prepared Host and both the Edge Node and Host use the same IP subnet/VLAN for the TEP interface.
• There is a port property called nestedTNConfig sent by the Management Plane (MP) to the host when the Edge TEP vNIC is attached to a NSX port group / Segment.
• This issue occurs when an Edge Node was deployed on a Host that was not prepared with NVDS host switch, and the nestedTNConfig property is not sent.
• Later the Host is prepared with an NVDS or VDS host switch and this causes hosts only listens to traffic for its TEP interface and drop all traffic for the TEP interface of the Edge Node.

Resolution

This issue is resolved in NSX-T versions 3.2.3 and 4.1.0 available at VMware Downloads.

Workaround:
There are three workarounds for this issue:

1. Put the TEP traffic of the Edge Node on a different VLAN than the host’s TEP VLAN.

• Create a new uplink profile with a new VLAN. If IP pool is used, create a new IP pool, which allocates IP addresses in the newly created VLAN.
• Apply this profile to the Edge Node.

Note: To implement this Workaround you will need a separate subnet/VLAN.
 

2. Create a new NVDS/VDS for the ESXi host and assign a different uplink (vmnic) to the NVDS/VDS.  Apply the same uplink profile (including VLAN) and IP pool to the new uplink.
 

Note: To implement this workaround, you will need additional vmnic(s) for the new NVDS/VDS

If you are unable to apply this workaround or it does not work for you, please open a support request with Broadcom support