Unable to Delete Objects Created on Global Manager – Objects Stuck in "Pending Deletion" State

Products

• VMware NSX

Issue/Introduction

• Objects created on Global Manager (GM) cannot be deleted.
  
  
• Objects may consist of Tier 0 gateway, Tier 1 gateway, Gateway firewall rules, Segments, Gateway firewall policies etc.
  
  
• A GET API call confirms the objects have the marked_for_delete=true flag set, Some example API's below : 
  • GET https://{{GM_VIP}}/global-manager/api/v1/global-infra/domains/default/gateway-policies?include_mark_for_delete_objects=true
  • GET https://{{GM_VIP}}/global-manager/api/v1/global-infra/segments?include_mark_for_delete_objects=true
  • GET https://{{GM_VIP}}/global-manager/api/v1/global-infra/tier-0s?include_mark_for_delete_objects=true
  • GET https://{{GM_VIP}}/global-manager/api/v1/global-infra/tier-1s?include_mark_for_delete_objects=true
  • GET https://{{GM_VIP}}/global-manager/api/v1/global-infra/domains/#####/groups?include_mark_for_delete_objects=true
  • GET https://{{GM_VIP}}/global-manager/api/v1/global-infra/services?include_mark_for_delete_objects=true
    
    
• Performing a DELETE API to any of these objects returns 200 OK, however the object still remains as Pending Deletion in the UI.

Note : Always verify the correct API path from the NSX API Guide

Imp Note : If you are experiencing an issue where NSX Segments created on the Local Manager are stuck in a 'marked_for_delete=true' state, please refer to KB article Removing NSX Segments that are stuck in "Marked for Delete" state

Environment

VMware NSX

Cause

• The deletion synchronization between the Global Manager and Local Manager becomes stuck because the deletion queue is blocked. This prevents the "Marked For Delete" status from being processed and cleared during the standard synchronization cycle. (i.e. Synchronization between GM and LM does not properly process delete operations).
  
  
• Every two hours, the system initiates a cleanup of the Traceflow Config. This process also removes the corresponding Traceflow Observations and Traceflow Status entries. During this cleanup, the system populates the cache on the GM to ensure that Traceflow Observations are sent only to the appropriate LMs. The expected outcome is that all observations are successfully cleaned up.
  
  
• However, when the queue size continues to increase, we observe that Traceflow Observations persist in the system even after the associated Traceflow Config and Traceflow Status have been deleted. This behavior causes inconsistencies in the GM cache, as the DELETE notifications from the LMs are not handled correctly by the GM.

Resolution

• If you are experiencing an issue where NSX Segments created on the Local Manager are stuck in a 'marked_for_delete=true' state, please refer to KB article Removing NSX Segments that are stuck in "Marked for Delete" state
• Run the below command from the root shell of the Global Managers to confirm how many objects are in marked_for_delete = true state.

/opt/vmware/bin/corfu_tool_runner.py -n nsx -o showTable -t DeletesPending

• If there are multiple objects marked_for_delete, follow the resolution mentioned in article >> NSX Global Manager Alarm "Queue Occupancy Threshold Exceeded"

If the issue persists, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.

 

Additional Information

NSX API Guide