Unable to delete the Tier-1 gateways through the NSX UI or API

Products

• VMware NSX

Issue/Introduction

• Unable to delete Tier-1 gateways through the NSX UI or API
• Error: The object cannot be deleted as either it has children or it is being referenced by other objects
• DELETE /policy/api/v1/infra/tier-1s/{tier-1-id} API fails with the same error 
• The T1 gateway does not have any visible connections with other objection in the nsx
• The issue is seen in environments with QOS profiles created
• /var/log/proton/nsxapi.log on the NSX manager node will report error similar to the snippet below
  
  

  2024-07-04T06:25:18.297Z  WARN http-nio-127.0.0.1-7440-exec-14939 TransactionRetryAspect 4895 - [nsx@6876 comp="nsx-manager" level="WARNING" reqId="4#######-####-####-####-############" subcomp="manager" username="admin"] void com.vmware.nsx.management.policy.connectivity.service.NetworkServiceImpl.delete(PolicyPath) failed with class com.vmware.nsx.management.policy.policyframework.exceptions.ObjectInUseException.
  2024-07-04T06:25:18.300Z  INFO http-nio-127.0.0.1-7440-exec-14939 NsxBaseRestController 4895 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Error in API /nsxapi/api/v1/infra/tier-1s/Tier-1-GW caused by exception com.vmware.nsx.management.policy.policyframework.exceptions.ObjectInUseException:  {"moduleName":"Policy","errorCode":500030,"errorMessage":"The object path=[/infra/tier-1s/Tier-1-GW] cannot be deleted as either it has children or it is being referenced by other objects path=[/infra/qos-profiles/qos-profile-5,/infra/qos-profiles/qos-profile-6,/infra/qos-profiles/1qos-profile-5,/infra/qos-profiles/qos-profile-4,/infra/qos-profiles/qos-profile-7,/infra/qos-profiles/qos-profile-11,/infra/qos-profiles/qos-profile-8,/infra/qos-profiles/qos-profile-9,/infra/qos-profiles/qos-profile-10,/infra/qos-profiles/qos-profile-1,/infra/qos-profiles/qos-profile-2,/infra/tier-1s/Tier-1-GW/locale-services/default,/infra/qos-profiles/qos-profile-12,/infra/qos-profiles/qos-profile-3]"}

Environment

VMware NSX 3.2.x 
VMware NSX 4.1.x

Cause

Stale Relationships between the Tier-1 gateway and QOS Profile is preventing the Tier-1 gateway deletion

Resolution

This is a known issue and engineering team is aware of it. The issue is resolved in NSX-T version 3.2.3 and 4.1.1

Note: The scripts attached to this KB are intended to be run on NSX version 3.2.x and 4.1.x only. For any other versions, please raise a Broadcom support ticket. 

NSX Version 3.2.1:

1. Take a back up of customers NSX env
2. Login as root user to one of the NSX manager
3. Copy 3.2.x-logical-migration.jar file to /tmp
4. Execute logical migration:
java -Dcorfu-property-file-path=/opt/vmware/proton-tomcat/conf/ufo-factory.properties -DserviceType=nsx-policy-manager -DFromVersion=3.2.1 -cp 3.2.x-logical-migration.jar com.vmware.nsx.management.migration.impl.LogicalMigration


NSX Version 4.1.0.2: 

1. Take a back up of NSX env
2. Login as root user to one of the NSX manager
3. Copy 4.1.x-logical-migration.jar file to /tmp
4. Execute logical migration:
java -Dcorfu-property-file-path=/opt/vmware/proton-tomcat/conf/ufo-factory.properties -DserviceType=nsx-policy-manager -DFromVersion=4.1.0 -cp 4.1.x-logical-migration.jar com.vmware.nsx.management.migration.impl.LogicalMigration


Note: If you see an exception with missing zstd-jni, run the below command
sudo mount /tmp -o remount,exec