NSX for vSphere 6.x VTEP and vDS Uplink dependencies

Products

• VMware NSX

Issue/Introduction

This article provides information discussing a number of considerations when developing a vDS design to be used with NSX connectivity.

Symptoms:
NSX VXLAN Tunnel End Point (VTEP) Interfaces are dependent upon the virtual Distributed Switch (vDS) uplink configuration and the Teaming Policy selected when a cluster is prepared for Logical Networking.

Environment

VMware NSX for vSphere 6.3.x
VMware NSX for vSphere 6.2.x

Resolution

The NSX Network Virtualization Design Guide discusses a number of considerations when developing a vDS design to be used with NSX connectivity. Some highlights and clarifications are noted here, to assist in implementing a suitable configuration.

• Although an NSX Transport Zone can span multiple clusters, as well as multiple vDS, all clusters attached to a vDS used for NSX should be included in the same Transport Zone (Transport Zone alignment). This is not mandatory if clusters outside the Transport Zone do not run NSX workloads.
• Transport Zones are not intended as Security Zones, and in most cases a single Transport Zone is sufficient. For designs that use multiple Transport Zones, VMware recommends to review this with VMware Support before implementation to ensure it is a suitable design.
• In any cluster, only one VDS can be prepared for NSX. Different clusters may use a different vDS that has been prepared for logical networking.
• Within a given cluster using a vDS prepared for NSX logical networking, all hosts must use the same uplink configuration and VLAN ID. The teaming policy for VXLAN traffic must be the same for a given vDS. Other, non-NSX, port groups may use a different teaming policy, except when using LACP, in which case all port groups must use LACP for a given vDS.
• When using a multi-VTEP teaming policy (Route Based on Originating Port or Route Based on Source MAC Hash) The number of active uplinks configured on the vDS must all be configured with NSX VTEPs. It is not possible to isolate uplinks to specific port groups in this case, as NSX will provision a VTEP vmkernel interface for every active uplink. To isolate uplinks for non-VXLAN traffic with a multi-VTEP configuration, a separate vDS should be provisioned and the non-VXLAN traffic moved to that vDS, or a single VTEP teaming policy can be used, and the correct uplinks set to active for the desired port groups.

Note: For simplicity, the separate vDS solution is desired, as NSX will create a new portgroup for every logical switch created, and will assign the uplink configuration that was in place when the cluster was originally provisioned for VXLAN.

• If using LACP or static etherchannel for uplink configuration, the LACP or Route Based on IP Hash teaming policy, respectively, must be used for the NSX logical network preparation. NSX will create a single VTEP and rely on the vDS flow-based load balancing to use all active members of the LAG. When using this type of configuration, all portgroups on the vDS must use this same uplink configuration and teaming policy for all the traffic types. An option to provide the additional flexibility of using different teaming or uplink configuration would be to use a separate vDS for those port-groups.
• In addition, when using LACP for NSX VTEPs, VMware recommends to select a load balancing method on both the vDS and the physical switch that takes into account the full L4 header of packets (specifically the SRC port) for determining the LAG member to use for a given flow, since the LACP will always be looking at VXLAN-encapsulated traffic, which will always be UDP, have the same DST port, the same SRC IP (that of the VTEP vmk) and the same VLAN. The SRC port however will have a random value based upon the original (unencapsulated) packet's L4 Header). To achieve better load distribution over the LAG members, on the vDS use the "Source and destination IP address, TCP/UDP port and VLAN" load balancing mode, and a similar method on the physical switch that takes all L4 packet information into account.
• Once prepared for NSX and TEP's configured , modifying the uplink numbers or Teaming policy from vCenter would not take effect unless we reconfigure NSX again. 

Note: NSX versions up to and including at least 6.3.2 do not support more than 1 LAG group on a vDS.