Static route synchronization failures due to stale Corfu configuration in NSX-T

Products

• VMware NSX

Issue/Introduction

This article provides guidance on resolving synchronization issues where static routes configured in the NSX-T Manager GUI are not realized in the dataplane on Edge transport nodes.

This behavior is typically caused by stale configurations stored in the Corfu database, which prevent the NSX-T RouteGraphManager from successfully processing updates for logical routers.

 

Symptoms

• Static routes configured in the NSX-T Manager GUI are not reflected in the Edge node CLI forwarding table.
• nsxapi.log reveals RouteGraphManager errors or NullPointerException during the processing of static route work-items.
  

  <timestamps> UTC ERROR workerTaskExecutor-1-6 EdgeWorkItemExceptionAspect 5093 - [nsx@6876 comp="nsx-manager" errorCode="MP11268" level="ERROR" subcomp="manager"] Failed to process work-items [WorkItem{identifier=StaticRoute/########-####-####-####-7f8a3803e7c5, Timestamp{epoch=159, address=<IP_ADDRESS>}}] of lr LogicalRouter/########-####-####-####-d86a51d8dae4 by method processStaticRouteWorkItems
  java.lang.NullPointerException: null
      at com.vmware.nsx.management.edge.lrouter.routing.rec.RouteGraphUtils.addEdgesToParentsAndChildren(RouteGraphUtils.java:269) ~[?:?]
      at com.vmware.nsx.management.edge.lrouter.routing.rec.RouteGraphUtils.addNhToRouteNode(RouteGraphUtils.java:65) ~[?:?]
      at com.vmware.nsx.management.edge.lrouter.routing.rec.RouteGraphManager.addNetworkForRoute(RouteGraphManager.java:183) ~[?:?]
      at com.vmware.nsx.management.edge.lrouter.routing.rec.RouteGraphManager.addNetworkForRoute(RouteGraphManager.java:100) ~[?:?]
      at com.vmware.nsx.management.edge.lrouter.routing.rec.RouteGraphManager.addAllStaticRoutes(RouteGraphManager.java:579) ~[?:?]
      at com.vmware.nsx.management.edge.lrouter.routing.rec.RouteGraphManager.buildGraph(RouteGraphManager.java:533) ~[?:?]

• Connectivity to the destination network remains down despite successful configuration of static route via NSX Manager GUI.
  

Environment

VMware NSX

Cause

• The route graph algorithm failed to resolve the next-hop reachability for the static route.
• This failure occurs because the manager treats the route as non-realizable due to a stale interface on the same gateway having overlapping subnets with another interface on the gateway.

Resolution

• Identify the stale interface/config:

  1. From any NSX Manager login via SSH using admin
     • run: get gateway
       • identify the SR UUID in question. 
  2. Run: get gateway <UUID> interface
     • Analyze to see if a stale interface is present. Below there are 2 interfaces with same name. The stale interface has overlapping subnet with another interface.
       
  3. Collect log bundles from all 3 NSX Manager nodes. 
  4. Collect a CORFU dump so all related entries can be reviewed/updated. The process to create and export a CORFU dump can be found below. 
  5. Once all the above data is collected, proceed with opening an SR with Broadcom NSX Support: Creating and managing Broadcom cases

 

• Gathering CORFU DB Dump:

1. 1. Use script attached (corfu_script.sh) to capture a CORFU dump from one of the NSX Managers.
      • You may wish to run it on a manager node that is NOT the leader / holder of the VIP.  
      • NOTE: It may take over an hour to complete.

   2. Upload the attached script (corfu_script.sh) file to NSX Manager

   3. Run: # chmod +x corfu_script.sh

   4. Run this corfu_script.sh file on manager node as # ./corfu-script.sh

      • The script actually reads and stores corfu tables in "corfuBackup" directory.

        • Attach all table data stored by script in "corfuBackup" directory.
        • You can first go to that directory, then use "tar -czvf corfuBackup.tar.gz ." to create an archive file with the contents of that directory for ease of transfer.