North-South Connectivity Fails When Upstream Edge is Configured with Trunk Interfaces
Article ID: 325134
Updated On:
Products
VMware NSX Networking
Issue/Introduction
Symptoms:
When NSX Edge interface is configured as trunk connecting (downlink) to multiple DLRs, North-South connectivity to VMs on some VNIs behind the DLR fails.
The Edge cluster is configured with two or more VTEPs.
When NSX Edge interface is configured as trunk connecting (downlink) to multiple DLRs, North-South connectivity to VMs on some VNIs behind the DLR fails.
The Edge cluster is configured with two or more VTEPs.
Environment
VMware NSX for vSphere 6.4.x
VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.3.x
Cause
The issue is due to the controller being unaware of all the VTEPs configured on the edge host, in particular the VTEP IP of the vdr-vdrport. The code path that handles TRUNK configuration only informs the controllers about one of the VTEPs and not that of the vdr-vdrPort mapped to the other VTEP.
When the host (where the VM resides) VDR ARPs for the next hop (ESG interface) over the transit VNI, it is sent only to one of the VTEP-IP/vmnicX of the Edge Host. Since the vdr-vdrPort on Edge host is mapped to a separate VTEP-IP/vmnicX, if the packet received on the wrong VTEP/uplink, it is dropped by the RPF check filter on the vswitch, resulting in ARP resolution failure , thereby disrupting S-N traffic towards the Edge.
When the host (where the VM resides) VDR ARPs for the next hop (ESG interface) over the transit VNI, it is sent only to one of the VTEP-IP/vmnicX of the Edge Host. Since the vdr-vdrPort on Edge host is mapped to a separate VTEP-IP/vmnicX, if the packet received on the wrong VTEP/uplink, it is dropped by the RPF check filter on the vswitch, resulting in ARP resolution failure , thereby disrupting S-N traffic towards the Edge.
Resolution
This issue is resolved in:
Workaround:
There are two workarounds available to remediate this issue:
1. On the ESG host, deploy dummy VMs with multiple interfaces attached to each of the DLR transit LS. Since we can only do 10 vNICs per VM, you will need to determine first how many dummy edge VMs will be needed. If you have 15 DLRs attached, then you will need to 2 dummy VMs. If you have 25, you will need three. No need to assign IP addresses to the interfaces. Connecting the interfaces directly to VMs on the Edge host will push both VTEP-IPs on the edge host to the controller.
Note: it could be any VM with ‘n’ vNICs based on requirement. VMware recommends the Compact size Edge VM as its easy to deploy and readily available from NSX, no need to look for OVA/images.
2. Remove the trunk configuration, deploy one more ESG and attach the DLR transit VNIs to individual interfaces on the the edges. Basically, eliminate the Trunk Interface architecture.
VMware NSX for vSphere 6.3.7, available at VMware Downloads.
VMware NSX for vSphere 6.4.4, available at VMware Downloads.
Workaround:
There are two workarounds available to remediate this issue:
1. On the ESG host, deploy dummy VMs with multiple interfaces attached to each of the DLR transit LS. Since we can only do 10 vNICs per VM, you will need to determine first how many dummy edge VMs will be needed. If you have 15 DLRs attached, then you will need to 2 dummy VMs. If you have 25, you will need three. No need to assign IP addresses to the interfaces. Connecting the interfaces directly to VMs on the Edge host will push both VTEP-IPs on the edge host to the controller.
Note: it could be any VM with ‘n’ vNICs based on requirement. VMware recommends the Compact size Edge VM as its easy to deploy and readily available from NSX, no need to look for OVA/images.
2. Remove the trunk configuration, deploy one more ESG and attach the DLR transit VNIs to individual interfaces on the the edges. Basically, eliminate the Trunk Interface architecture.
Feedback
Yes
No
Powered by
