NSX VPC connectivity issues with Private AI Services

Products

• VMware NSX

Issue/Introduction

• NSX 9.x is deployed with AVI Load Balancer.
• Connectivity from a Private VPC subnet with SNAT to an external machine fails.
• Capture on the Edge where T0 resides shows egress and ingress traffic but the ingress traffic does not return to the source ESX host.
• Traceflow command on T0 Edge shows traffic is dropped due to "Dropped due to IP failure".
  
  

  edge> traceflow <uuid> <base64 of packet>
Packet Trace Results
...

  EVENT                : logical_dropped
PACKET_BASE64        : <base64 of packet>
PARENT_UUID          : <UUID>
REASON               : Dropped due to IP failure
TIMESTAMP            : <Timestamp>
TRACE_UUID           : <UUID>
TYPE                 : logical_router_port
  
  

• T0 routing table shows duplicates routes for problematic subnet.
  
  

  edge(tier0_sr[10])> get route| more
<Timestamp>

  Flags: t0c - Tier0-Connected, t0s - Tier0-Static, b - BGP, o - OSPF
t0n - Tier0-NAT, t1s - Tier1-Static, t1c - Tier1-Connected,
t1n: Tier1-NAT, t1l: Tier1-LB VIP, t1ls: Tier1-LB SNAT,
t1d: Tier1-DNS FORWARDER, t1ipsec: Tier1-IPSec, isr: Inter-SR, isrs: Inter-SR-static
ivs: Inter-VRF-Static, tgws: Transit-Gateway-Static, > - selected route, * - FIB route
prs: Provider Static, is: Internal Static

  Total number of routes: <number>
...
tgws> * 100.#.#.1/32 [5/0] via 169.254.64.1, inter-rt-296, 4d01h50m  
tgws> * 100.#.#.1/32 [5/0] via 169.254.64.2, inter-rt-296, 4d01h50m  
  
  

Environment

VMware NSX 9.1.0.0

Cause

A VPC static route created for AVI Load Balancer has an advertised route on TGW/T0 which is not cleared from TGW and T0 on StaticRoute deletion. When the same IP is used for another service like NAT, there are two routes for the same IP causing north-south connectivity issues.

Resolution

This issue is resolved in VMware NSX 9.1.1, available at Broadcom downloads.

Workaround:

To workaround execute the following API against the NSX Manager, this will remove the duplicate route:

POST /policy/api/v1/orgs/<orgId>/projects/<projectId>/vpcs/<vpcId>/force-resync