VMs on the same ESXi host as Edge VMs experience North-South connectivity

Products

• VMware NSX

Issue/Introduction

The following conditions are met:

• NSX-T Data Center 3.1.0 or 3.1.1.
• vSphere 7.0.
• Edge and Host TEPs are connected to the same NVDS/VDS.
• Edge cluster includes Edges configured with different TEP subnets (from multiple sites for example).

In this scenario VMs on the same ESXi host as Edge VMs have no North-South connectivity and the T0/T1 DR fails to resolve the ARP for the Transit IP.

#net-vdr -N -l 301f4d66-####-####-####-####
DR 301f4d66-####-####-####-#### ARP Information :

Network           Mac                  Flags      Expiry     SrcPort    Refcnt     Lif UUID
-------           ---                  -----      ------     -------    ------     ---------
169.254.0.2      00:00:00:00:00:00    N                     0          2          ########-####-####-####-########8972
169.254.0.1      ##:##:##:##:##:52    VI         permanent  0          1          ########-####-####-####-########8972
10.10.10.5       ##:##:##:##:##:6c    VL         461        67108961   14         ########-####-####-####-########03e2

Environment

VMware NSX-T
VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

The ARP resolution failure is caused by an issue with the hierarchical two-tier(MTEP) replication when VMs are running on the same ESXi hostswitch as Edge VMs and Edge TEP subnet is the same as the Host TEP subnet.

Resolution

Currently, there is no resolution.

Workaround:
Currently there are two ways to work around this issue:

1. Use a different subnet for Edge and Host TEPs.

2. Configure an additional NVDS/VDS (requires two extra vmnics) and connect the Edge vNics to it.

     [Edge VMs]               [Overlay workload VMs]
         |                               |
         |                               |
    [ NVDS/VDS ]                    [ NVDS/VDS ]
         |                               |
         |                               |
     vmnic1,2                         vmnic3,4