North-South Connectivity Fails When Upstream Edge is Configured with Trunk Interfaces

Products

• VMware NSX

Issue/Introduction

When NSX Edge interface is configured as trunk connecting (downlink) to multiple DLRs, North-South connectivity to VMs on some VNIs behind the DLR fails.

Environment

VMware NSX for vSphere 6.x

 

The NSX Edge cluster is configured with two or more VTEPs.

Cause

The issue is due to the controller being unaware of all the VTEPs configured on the edge host, in particular the VTEP IP of the vdr-vdrport. The code path that handles TRUNK configuration only informs the controllers about one of the VTEPs and not that of the vdr-vdrPort mapped to the other VTEP. 

When the host (where the VM resides) VDR ARPs for the next hop (ESG interface) over the transit VNI, it is sent only to one of the VTEP-IP/vmnicX of the Edge Host. Since the vdr-vdrPort on Edge host is mapped to a separate VTEP-IP/vmnicX, if the packet received on the wrong VTEP/uplink, it is dropped by the RPF check filter on the vswitch, resulting in ARP resolution failure , thereby disrupting S-N traffic towards the Edge.

Resolution

This issue is resolved in VMware NSX for vSphere 6.4.4

There are two workarounds available to remediate this issue:

1. On the ESG host, deploy dummy VMs with multiple interfaces attached to each of the DLR transit LS. Since only 10 vNICs per VM can be configured, you will need to determine first how many dummy edge VMs will be needed. If there are 15 DLRs attached, then you will need to 2 dummy VMs. If you have 25, you will need three. There is no need to assign IP addresses to the interfaces. Connecting the interfaces directly to VMs on the Edge host will push both VTEP-IPs on the edge host to the controller.
   
   Note: it could be any VM with ‘n’ vNICs based on requirement. VMware recommends the Compact size Edge VM as its easy to deploy and readily available from NSX.
   
   
2. Remove the trunk configuration, deploy one more ESG and attach the DLR transit VNIs to individual interfaces on the the edges