Unable to delete the Tier-1 gateways through the NSX UI or API

Products

• VMware NSX

Issue/Introduction

• Unable to delete Tier-1 gateways through the NSX UI or API
• Error: The object cannot be deleted as either it has children or it is being referenced by other objects
• DELETE /policy/api/v1/infra/tier-1s/{tier-1-id} API fails with the same error 
• The T1 gateway does not have any references or links with other objects in the NSX
• The issue is seen in environments with QOS profiles created at some point in time, and may not be in use anymore
• /var/log/proton/nsxapi.log on the NSX manager node will report error similar to the snippet below
  

  2024-07-04T06:25:18.297Z  WARN http-nio-127.0.0.1-7440-exec-14939 TransactionRetryAspect 4895 - [nsx@6876 comp="nsx-manager" level="WARNING" reqId="4#######-####-####-####-############" subcomp="manager" username="admin"] void com.vmware.nsx.management.policy.connectivity.service.NetworkServiceImpl.delete(PolicyPath) failed with class com.vmware.nsx.management.policy.policyframework.exceptions.ObjectInUseException.
  2024-07-04T06:25:18.300Z  INFO http-nio-127.0.0.1-7440-exec-14939 NsxBaseRestController 4895 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/infra/tier-1s/Tier-1-GW caused by exception com.vmware.nsx.management.policy.policyframework.exceptions.ObjectInUseException:  {"moduleName":"Policy","errorCode":500030,"errorMessage":"The object path=[/infra/tier-1s/Tier-1-GW] cannot be deleted as either it has children or it is being referenced by other objects path=[/infra/qos-profiles/qos-profile-5,/infra/qos-profiles/qos-profile-6,/infra/qos-profiles/1qos-profile-5,/infra/qos-profiles/qos-profile-4,/infra/qos-profiles/qos-profile-7,/infra/qos-profiles/qos-profile-11,/infra/qos-profiles/qos-profile-8,/infra/qos-profiles/qos-profile-9,/infra/qos-profiles/qos-profile-10,/infra/qos-profiles/qos-profile-1,/infra/qos-profiles/qos-profile-2,/infra/tier-1s/Tier-1-GW/locale-services/default,/infra/qos-profiles/qos-profile-12,/infra/qos-profiles/qos-profile-3]"}

Environment

VMware NSX 3.2.x 
VMware NSX 4.1.x

Cause

Stale Relationships between the Tier-1 gateway and QOS Profile is preventing the Tier-1 gateway deletion

Resolution

This issue is resolved in VMware NSX-T Data Center 3.2.3 and VMware NSX 4.1.1, 4.2.0 and above, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Note: The scripts attached to this KB are intended to be run on NSX version 3.2.x and 4.1.x only. For any other versions, please raise a Broadcom support ticket. 

NSX Version 3.2.1:

1. Complete a backup of the NSX Manager. 
2. Login as root user to one of the NSX manager
3. Copy 3.2.x-logical-migration.jar file to /tmp
4. Execute logical migration:
java -Dcorfu-property-file-path=/opt/vmware/proton-tomcat/conf/ufo-factory.properties -DserviceType=nsx-policy-manager -DFromVersion=3.2.1 -cp 3.2.x-logical-migration.jar com.vmware.nsx.management.migration.impl.LogicalMigration


NSX Version 4.1.0.2: 

1. Complete a backup of the NSX Manager. 
2. Login as root user to one of the NSX manager
3. Copy 4.1.x-logical-migration.jar file to /tmp
4. Execute logical migration:
java -Dcorfu-property-file-path=/opt/vmware/proton-tomcat/conf/ufo-factory.properties -DserviceType=nsx-policy-manager -DFromVersion=4.1.0 -cp 4.1.x-logical-migration.jar com.vmware.nsx.management.migration.impl.LogicalMigration


Note: If you see an exception with missing zstd-jni, run the below command
sudo mount /tmp -o remount,exec