T0 VRF deletion fails and VRF shows as in progress state after using automation

Products

• VMware NSX

Issue/Introduction

• You are unable to delete a Tier-0 VRF Gateway through the NSX-T UI or API.
• The delete options in the UI may appear greyed out.
• The initial deletion workflow was executed via automation (API or script) rather than the NSX-T UI, this workflow involved deleting child objects of the VRF (Interfaces and HA_VIP configuration) followed by the VRF itself.
• The T0-VRF shows as "in progress" status.
  
  
• In the NSX Manager logs (/var/log/nsx/nsxapi.log or syslog), you see errors similar to the following:
  [nsx@6876 comp="nsx-manager" errorCode="PM500016" level="ERROR" subcomp="manager"] Unable to delete provider /infra/tier-0s/<VRF-ID>
[nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Message returned [Routing] This logical router port LrPort/<UUID> has ha vip configured. It cannot be deleted, please remove the ha vip configuration from LogicalRouter LogicalRouter/<UUID> first.

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX

Cause

This issue occurs due to a race condition triggered by a specific automation workflow where two operations to the VRF configuration are executed in quick succession without a pause.

Because the steps occur back-to-back, the updates overlap. The process responsible for cleaning up the HA VIP configuration fails because the VRF enters a "delete pending" state from the subsequent deletion calls. Consequently, the HA VIP configuration remains on the port, creating a dependency that prevents the Tier-0 VRF from being deleted.

Resolution

This is a known issue impacting VMware NSX. 

Workaround

If you believe you have encountered this issue and have a VRF already in this state, open a support case with Broadcom Support and refer to this KB article. For more information, see Creating and managing Broadcom support cases.

Additional Information

To prevent this issue in future automation scripts, ensure that the script verifies the realization of the PATCH or delete operation (removal of HA VIP) before initiating the DELETE operation for the Tier-0 or its interfaces or simply add a 1 second pause in between each delete call.