Uplink2 on an Edge VM used by a Tier-0 GW will not forward VLAN traffic when named teaming policies have not been set on the VLAN-backed Segments connecting the T0 GW uplinks to the physical underlay.

Products

• VMware NSX

Issue/Introduction

• VLAN traffic between the T0 GW and the physical underlay will not be forwarded in either direction using uplink2 of the Edge(s) in question.
• BGP peering to an upstream neighbor will not establish from uplink2.
• Named teaming policies specifying a single uplink as being active may have been configured in the Edge uplink profile and may have been added to the VLAN TZ used by the Edges, but they have not been configured on the VLAN segments being used to connect the T0 GW to the physical underlay.  See below where it says "Not Set"
  
  
  
• Since the named teaming policies were not added to the VLAN segments, the default teaming policy will automatically be chosen, as it is the only teaming policy available.  Default teaming = LOADBALANCE_SRID with uplink1 and uplink2 BOTH being listed as active (In this order, with Uplink1 listed first).
• The Tier-0 GW could be configured in active-active or active-standby mode. 
• Running the below CLI command on the Edge and then examining the interfaces marked as uplink-1 and uplink-2 will reveal that the uplink-2 interface has been assigned the same MAC address as the uplink-1 interface:
  
  nsx-edge-1> get logical-router <LR_UUID> interfaces
  
  Interface     : 
      Ifuid         : 
      Name          :  uplink-1
      Fwd-mode      :
      Internal name : 
      Mode          : 
      Port-type     : uplink
      IP/Mask       : 
      MAC           : 00:50:56:AA:BB:CC <----  Same MAC address as below
      VLAN          : 
      Access-VLAN   :
      LS port       : 
      Urpf-mode     : 
      DAD-mode      : 
      RA-mode       : 
      Admin         : up
      Op_state      : up
      Enable-mcast  : 
      MTU           : 
      arp_proxy     :
  
  Interface     : 
      Ifuid         : 
      Name          : uplink-2
      Fwd-mode      :
      Internal name : 
      Mode          : 
      Port-type     : uplink
      IP/Mask       : 
      MAC           : 00:50:56:AA:BB:CC
      VLAN          : 
      Access-VLAN   :
      LS port       : 
      Urpf-mode     : 
      DAD-mode      : 
      RA-mode       : 
      Admin         : up
      Op_state      : up
      Enable-mcast  : 
      MTU           : 
      arp_proxy     :

Environment

VMware NSX

Cause

This is not an actual issue, as it is working as expected.

• The default teaming policy that is automatically added to an uplink profile used by an edge is actually intended to be used by the Overlay TEPs, and not by the Tier-0 uplinks connecting it to the physical underlay (In fact, the Overlay TEPs will always use the default teaming policy, regardless as to whether named teaming policies have been configured in the uplink profile or not).
• When a named teaming policy has not been configured within the Edge's uplink profile, added to the VLAN TZ and lastly, configured within the individual VLAN segments, the VLAN traffic appears to attempt to use the default teaming policy, which specifies that both uplinks are active. 
  • It looks at the default teaming policy, since it is the only teaming policy available.  It sees uplink-1 listed before uplink-2 in the "Active Uplinks" column.  See below: 
    
  • At this point, the newly created uplinks BOTH choose to take the MAC address assigned to the Edges fp-eth0 interface as their own, simply because uplink-1 is listed first in the column, as seen above.  When using the supported method described earlier for the dual uplink VLAN traffic, only uplink1 would receive the MAC address used by the Edge's fp-eth0 interface, while uplink2 would receive and use the MAC address assigned to the Edge's fp-eth1 interface.
  • At this point, if the Edge's fp-eth0 interface had been connected to a VLAN segment or DPG that permits one of the VLANs used by the T0 GWs to connect to the physical underlay (but just one), while the fp-eth1 interface is connected to a VLAN segment or DPG that permits the other VLAN, then uplink1 would forward traffic on the VLAN used by the VLAN segment that had the named teaming policy specifying uplink-1 as being active, while uplink2 would not forward traffic or establish BGP peering at all. (usually the networks that fp-eth0 and fp-eth1 connect to is configured during the initial edge deployment.  This information can be checked through the Edge details page in the NSX UI or by looking at the network configuration page after selecting the individual Edge in the vSphere client )
  • In the particular scenario that I worked on, the edges were connected to two different trunked distributed portgroups, both of which were set with a VLAN range of 0-4094 when the edges were first deployed.  As a result, the two DPG's each permitted both of the VLANs that had been configured on the two separate VLAN segments.  This resulted in some of the edges seeing both uplinks being assigned the same MAC address, since they both came from fp-eth0, but still appearing to be forwarding traffic and peering with BGP neighbors as they normally would.  In actuality, the uplink1 interface was forwarding traffic between the Tier-0 GW and both upstream neighbors because it was using one of the VLANs to forward traffic to the first neighbor, and the second VLAN to forward traffic to the other neighbor.  (since the DPG's permitted both VLANs).

Resolution

• If it is desired that Edge uplink1 and uplink2 both forward Tier-0 VLAN traffic to and from the physical underlay, the below steps taken together constitute the supported method for making this work:
  • Create two (at least, depending on the design) named teaming policies in the uplink profile used by the Edges.  The names chosen should usually include the name of one of the uplinks to avoid confusion.  EX....  uplink-1-active, uplink-2-active.
  • The uplink-1-active named teaming policy should specify uplink1 as being active and not set uplink2 as being an active uplink OR a standby uplink.  
  • The uplink-2-active named teaming policy should specify uplink2 as being active and not set uplink1 as being an active uplink OR a standby uplink.  (The reason for this is that during an event that causes the active uplink to go down, an Edge VM does not have the ability to actually failover from the uplink set as active to an uplink set as standby.  This is by design.  High availability comes from the use of multiple Edges, not from a failover ability between uplinks on the same Edge VM)
  • Add both of the newly created named teaming policies to the VLAN TZ used by the Edges.
  • Configure/set the two named teaming policies on the VLAN segments the Edge will use to forward traffic between the T0 GW uplinks and the physical underlay.  The uplink-1-active named teaming policy should be set on the VLAN segment that is intended to use uplink1.  The uplink-2-active named teaming policy should be set on the VLAN segment that is intended to use uplink2.   Usually, the VLAN segments will be configured with different VLANs to match how the TOR and the rest of the physical underlay  has been configured.  Ensure that the appropriate named teaming policy is set on the individual VLAN segments.
  • At this point, uplink1 should be able to forward traffic in both directions between the Tier-0 GW and the physical underlay on one VLAN, while uplink2 should successfully forward traffic in the same manner as uplink1, but using the other VLAN.