Traffic Failure to Physical Hosts within Same Subnet as NSX Overlay Segment

Products

• VMware NSX

Issue/Introduction

VMs residing on an NSX Overlay Segment cannot reach specific physical IP addresses on the same IP subnet, but on a VLAN-backed physical network.

Symptoms:

• ICMP and TCP traffic from Overlay VMs to specific physical IPs (e.g., 10.##.##.##/32, 10.##.##.##/32) fails.

• The Overlay Segment and Physical VLAN share the same CIDR (e.g., 10.##.##.##/24).

• Traceroute shows traffic stopping at the Tier-1 Gateway (T1).

• Northbound BGP on the Tier-0 (T0) learns the specific /32 routes, but connectivity remains broken.

Environment

VMware NSX

Cause

The issue is caused by Routing Ambiguity at the Tier-1 Gateway. In a multi-tier NSX topology, the T1 Gateway holds a "Connected Route" for its Overlay Segment. This connected route has a higher priority than routes learned from the Tier-0. Because the T1 does not typically ingest specific /32 routes from the T0 to override its local interface, it assumes all traffic for the 10.##.##.##/24 range is local to the Overlay. Traffic destined for physical hosts in that same range is blackholed because the T1 cannot find those MAC addresses on the Overlay and refuses to route the traffic Northbound.

Resolution

There are three primary methods to resolve this conflict:

Option 1: Migrate to a VLAN-backed Segment (Recommended if Overlay is not required)

1. Create a new VLAN-backed Segment using the existing physical VLAN ID.

2. Migrate the VM vNICs from the Overlay Segment to the new VLAN-backed Segment.

3. Decommission the conflicting Overlay Segment.

• Result: VMs communicate at Layer 2 via ARP, bypassing the T1 routing conflict.

Option 2: Implement NSX Layer 2 Bridging (Best for Subnet Extension)

Extend the Layer 2 broadcast domain from the physical VLAN into the NSX Overlay.

1. Create an Edge Bridge Profile.

2. Assign the Bridge Profile to the Overlay Segment and map it to the physical VLAN.

• Result: ARP requests broadcast across both environments, allowing Overlay VMs to discover physical hosts on the same subnet.

Option 3: Re-IP the Overlay Segment

1. Change the IP gateway and member addresses of the NSX Overlay to a unique subnet (e.g., 10.##.##.##/24).

• Result: Removes the prefix overlap, allowing standard Longest Prefix Match (LPM) routing to function correctly.

Additional Information

For detailed steps on configuring an Edge Bridge to extend your segment, refer to the official documentation:

Edge Bridging: Extending Overlay Segments to VLAN