Addressing Common NSX Underlying Infrastructure Connectivity Issues

Products

• VMware NSX

Issue/Introduction

Inconsistent configurations in the underlying network infrastructure can cause connectivity issues. The goal of this document is to provide a checklist to easily identify and fix these configuration issues.

Environment

VMware NSX-T 3.x
VMware NSX-T 2.5.x
VMware NSX 4.x

Resolution

Checklist

1. VLAN config should match on segment/Edge logical uplinks, DVS uplinks, TOR
   1. Check VLAN config on the TOR connected to the physical NIC of the DVS uplink which provides connectivity to the Tier0 uplink
   2. Check DVS uplink config: Configure VLAN Tagging on a Distributed Port Group or Distributed Port  Usually the DVS port group corresponding to Tier0 uplinks has a trunk range as multiple uplinks with different VLANs are often pinned to the same edge nic.
   3. Check NSX Edge Tier0 uplink VLAN: Create a Tier-0 Uplink Segment

 

 

2. Consistent MTUs setting on LRPs (logical router ports), VDS, TOR
   1. Check MTU setting on the TOR interface connected to the physical NIC of the DVS uplink which provides connectivity to the Tier0 uplink
   2. VDS MTU setting: Enable Jumbo Frames on a vSphere Distributed Switch
   3. NSX MTU guidance: Guidance to Set Maximum Transmission Unit

 

3. RPF setting on NSX Edge uplinks
   1. Ideally,  asymmetric routing/forwarding is not recommended and therefore RPF is set to Strict mode on Tier0 uplinks.
   2. For symmetric routing configuration, please ensure that the configuration between Tier 0 and northbound routers is such that the same set of prefixes are advertised from each edge node within a Tier0 Gateway on a given site. Also, the same set of prefixes must be learned from TORs on all Edge nodes of a Tier0 Gateway on a given site.
   3. Asymmetric routing can happen when different BGP filters are applied on different nodes towards northbound neighbors in either direction leading to different routes getting advertised from different Edge nodes or if Edge nodes of a Tier0 Gateway have different BGP neighbors and learn/advertise different prefixes. Please check the BGP route maps to ensure that this is not the case.   
   4. For Federation environments with Primary/Secondary site design, advertise longer AS paths for BGP advertisements on secondary site BGP neighbors to resolve asymmetric forwarding.
   5. After evaluating the above mentioned, if you still need asymmetric forwarding in your environment, please change the Tier0 uplink RPF setting to None.

4. TEP VLAN mismatch
   1. Ensure that the VLAN configured on TOR matches the VLAN configured on the NSX side for overlay traffic.

5. Federation RTEP VLAN mismatch
   1. Ensure consistent VLAN setting for RTEP and the VLAN ID on the underlay router connected to this RTEP: Configure Edge Nodes for Stretched Networking

5. Edge connectivity issues after vmotion
   1. Set DRS rules as described in the following KB article: NSX-T Edge vMotion best practices (88034)
   2. Ensure VLAN/MTU config on TORs where the Edge may be vMotioned is consistent. Define the DRS vm/host rules such that the underlying hosts are on the same L2 domain and that the BGP/OSPF neighbors or static route next hops defined on Tier0 are reachable from all the hosts where the Edge may be vmotioned.