VM communication fails via service interface due to overlapping downlink static route

Products

VMware NSX

Issue/Introduction

Introduction

In VMware NSX environments, virtual machines connected to an overlay segment may experience traffic loss when attempting to reach destinations via a specific static route.
This issue occurs when a static route is configured with a next hop reachable through a service interface, but a broader overlapping static route exists with a next hop reached via a downlink.
Because the Distributed Router (DR) lacks awareness of the service interface, it prioritizes the broader route available on the downlink, preventing the packet from ever reaching the Service Router (SR) where the more specific route is programmed.

Symptom

VMs on an overlay segment are unable to communicate with destinations using static route configured on Tier-0/Tier-1 gateway.
The next hop of the static route is reached via a service interface of the gateway.
Traffic follows an unexpected path (e.g., misrouted to a larger overlapping subnet) instead of the intended Service Interface.
- For example, static routes below is configured;
  - Subnet 10.10.10.0/24 Next hop <IP reachable from a service interface>
  - Subnet 10.0.0.1/8 Next hop <IP reachable from a downlink connected to the gateway>
- VMs connected to an overlay segment can not reach 10.10.10.0/24.
- traceroute to 10.10.10.0/24 shows the packet is forwarded to the next hop for 10.0.0.1/8.
Running get forwarding on the NSX Edge shows both route is present on the Service Router (SR).
edge01> get logical-router
<Timestamp>
Logical Router
UUID VRF LR-ID Name Type Ports Neighbors
<UUID> 0 0 TUNNEL 4 6/5000 ...
<UUID> 5 22 SR-StaticRouteTestT1 SERVICE_ROUTER_TIER1 6 2/50000
<UUID> 6 21 DR-StaticRouteTestT1 DISTRIBUTED_ROUTER_TIER1 4 0/50000
edge01> vrf 5
edge01(tier1_sr[5])> get forwarding
<Timestamp>
Logical Router
UUID VRF LR-ID Name Type
<UUID> 5 22 SR-StaticRouteTestT1 SERVICE_ROUTER_TIER1
IPv4 Forwarding Table
IP Prefix Gateway IP Type UUID Gateway MAC
0.0.0.0/0 100.64.0.2 route <UUID>
10.0.0.0/8 10.1.0.1 route <UUID> <<<<<< Larger static route.
10.1.0.0/16 route <UUID>
10.1.0.54/32 route <UUID>
10.10.10.0/24 192.168.102.65 route <UUID> <<<<<< Problem static route is programmed on SR. The next hop is reachable from a service interface.
100.64.0.2/31 route <UUID>
100.64.0.3/32 route <UUID>
127.0.0.1/32 route <UUID>
169.254.0.0/28 route <UUID>
169.254.0.1/32 route <UUID>
169.254.0.2/32 route <UUID>
192.168.102.64/28 route <UUID>
192.168.102.78/32 route <UUID>
Running net-vdr -R -l <DR UUID> on the ESXi host shows the problem route is missing from the Distributed Router (DR).
DR UUID can be identified by get logical-router on Edge.
[root@esx:~] net-vdr -R -l <UUID>
DR <UUID> Route Table
Legend: [U: Up], [G: Gateway], [C: Connected], [I: Interface]
Legend: [H: Host], [B: Blackhole], [F: Soft Flush] [!: Reject] [E: ECMP]
Destination GenMask Gateway Flags Ref UpTime HitCount Lif UUID
----------- ------- ------- ----- --- ------ -------- ---------
0.0.0.0 0.0.0.0 169.254.0.2 UG 1 18038 5 <UUID> <<<<<< Default route to SR.
10.0.0.0 255.0.0.0 10.1.0.1 UG 1 18038 6 <UUID> <<<<<< Only the larger static route of which the next hop is reachable from DR is programmed.
10.1.0.0 255.255.0.0 0.0.0.0 UCI 1 18038 6 <UUID>
169.254.0.0 255.255.255.240 0.0.0.0 UCI 1 18038 6 <UUID>

Environment

VMware NSX 4.x

VMware NSX 9.x

Cause

This is a product limitation.

When a static route is configured on a Gateway with the next hop reached via a service interface, the route is only realized on the Service Router (SR) because the interface is available only on SR.
It is not injected into the Distributed Router (DR) routing table on the ESXi hosts.

Usually DR has the default route to its SR and packets are routed to the SR accordingly.
However, if there is a larger overlapping prefix on the DR, the DR forwards the traffic based on the static route, and the traffic never reaches the SR.

Resolution

Currently there is no resolution.

To workaround the issue, follow either workaround.

Method 1: Dual Next-Hop

The workaround is only applicable to Tier-1 gateways connected to a Tier-0 gateway.

Route to SR is injected to DR if the next hop is Tier-0.
Configure such routes with higher AD so that the route is realized only on DR.

Add a second next-hop to the static route to force realization on the DR:

Navigate to Networking > Tier-1 Gateways > [Gateway] > Static Routes.
Edit the problematic route.
Keep the existing next-hop.
Add a second Next Hop:
- IP Address: Use the IP of the connected Tier-0 Router-Link port (the transit subnet IP).
- Admin Distance: Set this to a higher value than the primary hop (e.g., if primary is 1, set this to 2).
Save the configuration.

Warning: Configure again the static route entry with higher AD after you disconnect the Tier-1 gateway from Tier-0 gateway and connect it again.

Even if the transit IP does not change, the route is not injected to ESXi DR although the static route entry is left as it was.
You need to configure the static route again to inject the route to ESXi DR; e.g. Change its name.
If the transit IP changes, you need to change the next hop accordingly.

Method 2: Use Service Interface

If the architecture allows, ensure the traffic enters the Tier-1 Gateway via a service interface rather than a downlink, as traffic already on the SR will respect the SR-only static route.

Disconnect the overlay segment from the gateway.
Create a service interface on the overlay segment.