NSX Tier-1 Static Routes stuck in "In Progress" or "Marked for Deletion" state

Products

• VMware NSX

Issue/Introduction

 

• Operational Impact: AVI Load Balancer VIPs may be unreachable or fail to provision. Attempts to change clouds (e.g., Green to Blue) or attach Data IPs to Service Engines fail.

• UI Observations:

  • In the NSX Manager UI (Networking ⇒ Tier-1 Gateways ⇒ <Select T-1> ⇒ Static Routes), routes originating from the AVI Controller appear with a status of "In Progress" or "Success" but remain greyed out and cannot be modified or deleted.
    

  • When attempting to modify or delete these routes, the following error is displayed:

    "Reason: An object with the same path=[/infra/tier-1s/Site/static-routes/...] is marked for deletion. Either use another path or wait for the purge cycle (max 5 minutes) for permanent removal of the object."

• Log Analysis:

  • The NSX Manager logs (/var/log/proton/nsxapi.log or /var/log/proton/syslog) show a contradiction where the object is reported as "Not Found" during a retrieval attempt (GET), but "Marked for Deletion" during a modification attempt (PATCH).

  • Error Code: PM500012

  Example Log Snippet:

  2026-02-09T20:03:07.129Z ... errorCode="PM500012" ... Entity /infra/tier-1s/.../StaticRouteName not found
...
2026-02-09T20:03:07.154Z ... Operation status: 'failure' Error: An object with the same path=[/infra/tier-1s/...] is marked for deletion. Either use another path or wait for the purge cycle (max 5 minutes) for permanent removal of the object.
  

  If you suspect this issue is occurring but cannot immediately reboot, you can verify the "Marked for Delete" state using the NSX Policy API.

  
  API Call to identify stuck objects:
  Perform a GET request on the Tier-1 static routes, specifically adding the query parameter include_mark_for_delete_objects=true.
  GET https://<nsx-manager-ip>/policy/api/v1/infra/tier-1s/<tier-1-id>/static-routes?include_mark_for_delete_objects=true

  If the response returns the route with marked_for_delete: true and it has persisted longer than the stated 5-minute purge cycle, apply the Resolution steps listed in the Resolution section below.

 

Environment

VMware NSX 4.2.x 

Cause

This issue is caused by a race condition or a stale state within the NSX Policy Manager database. The specific static route object has been flagged for deletion, but the background "purge cycle" (responsible for permanently removing the object) has failed to complete.

As a result, the object enters a "zombie" or "tombstoned" state:

1. It is invisible to standard GET requests (hence "Entity not found").

2. It still occupies the path namespace, blocking new PATCH or PUT requests (hence "Marked for deletion").

Resolution

To resolve this issue, the stuck processes holding the object in the "pending" state must be cleared.

Valid Resolution Steps:

1. Reboot NSX Managers:

   • Perform a sequential reboot of the NSX Manager cluster nodes. This forces a re-initialization of the Policy and Management plane services and clears ephemeral runtime states.

2. Reboot Edge Nodes:

   • If the issue persists or if data-plane connectivity remains inconsistent for the VIPs, reboot the associated NSX Edge Nodes.

 

Additional Information

Related Knowledge Base Articles

• Broadcom KB 407135: Stale static route exists on Tier-0 Gateway of Global Manager.
• Broadcom KB 322418: Promotion of Manager objects to Policy objects in NSX-T fails: "wait for the purge cycle".