In NSX-T Federation or Local Manager environment, static routes may persist in the dataplane even after they have been deleted or updated in the Global Manager (GM) or Local Manager (LM) UI

Products

• VMware NSX

Issue/Introduction

• A Tier-0 or Tier-1 gateway forwards traffic to an old or incorrect next-hop IP (e.g., an decommissioned AVI Service Engine IP).
•  NSX Traceflow shows traffic being delivered to a next-hop that does not exist in the current static route configuration.
• Discrepancies between the intended configuration (API/UI) and the realized state on the ESXi transport nodes.
  
  

To confirm a sharding failure or an in-memory route graph synchronization issue, review the nsxapi.log (proton) on the NSX Manager nodes.

1. Identify the Stale Next-Hop

Search for the affected network prefix in /var/log/proton/nsxapi.log. Look for the RouteGraphManager entries which dump the in-memory state of the routing table.

Example Log Entry showing multiple next-hops (one correct, two stale):

/var/log/nsxapi.log

<Time stamp> INFO workerTaskExecutor-1-46 RouteGraphManager 5201 ROUTING [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Affected networks from route graph are : 
[{
  Network : 53.#.#.#/32,   
  Direction : SOUTH_BOUND, 
  nhToRealExitNextHopMap : { 
    { nhIp: 10.#.#.35, static route id : StaticRoute/1834557b..., isDeleted: false } : [10.#.#.35], 
    { nhIp: 10.#.#.36, static route id : StaticRoute/1834557b..., isDeleted: false } : [10.#.#.36], 
    { nhIp: 10.#.#.34, static route id : StaticRoute/e792a356..., isDeleted: false } : [10.#.#.34]
  }
}]

 

In this example, .35 and .36 are stale 'ghost' routes that were previously deleted but remain marked as isDeleted: false in the graph cache.

2. Check Sharding Ownership

If the realization is failing, check if the current node believes it has the authority to process the route. Search for canCurrentNodeProcess in the logs:

<Time stamp>INFO policyProviderTaskScheduler-9 WorkerShardManager 5388 POLICY [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] canCurrentNodeProcess : false, for worker : LogicalRouterWorker, for workItem : StaticRoute/e792a356-####

If all nodes return false, this indicates a clustering or worker framework failure .

Environment

VMware NSX

Cause

This issue is typically caused by a failure in the synchronization process between the Management Plane nodes, where a leadership change or sync error (GM-to-LM) prevents the isDeleted: true flag from propagating to the in-memory route graph on specific sharded nodes.

Resolution

This issue is resolved in VMware NSX  4.1.2.4 and  NSX 4.2.0, available at Broadcom downloads.

If you are having difficulty finding and downloading software, review the Download Broadcom products and software KB. 

Workaround:

proton restart on all three managers node

From NSX manager root shell, need to run below command for proton restart 
/etc/init.d/proton restart