Workaround for Deleting Stuck PrefixList/Tier0/RouteMap Post-Upgrade from 3.1.x to 3.2.1+

Products

• VMware NSX Networking

Issue/Introduction

This KB provides a Workaround for an issue where the PrefixList, Tier0, or RouteMap gets stuck while deletion if the environment is upgraded from 3.1.x version to 3.2.1 or later version.

Symptoms:

• The issue is identified by the error message "One or more relationships exist for object with id" when attempting to delete Tier0, Prefix List, or Route Map.
• in the UI the object is greyed out as shown below

• The relevant logs can be found at /var/log/proton/nsxapi.log.

Example-1
/var/log/proton/nsxapi.log:YYYY-MM-DDT09:45:03.538Z ERROR providerTaskExecutor-2 PolicyProviderUtil 19231 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM0" level="ERROR" subcomp="manager"] Created alarm Alarm [policyPath=/infra/realized-state/enforcement-points/default/tier-0-logical-routers/abc01eocb0006415aab01/prefix-lists/acbouttztfd01/alarms/UUID, message=One or more relationships exist for object with id InternalPrefixListConfig/UUID1 from bgpNeighbor/UUID3,BgpNeighbor/UUID4

Example-2
/var/log/proton/nsxapi.log:YYYY-MM-DDT15:05:00.810Z nsxmgr140101-stg NSX 7447 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM0" level="ERROR" subcomp="manager"] Created alarm Alarm [policyPath=/infra/realized-state/enforcement-points/default/tier-0-logical-routers/tier0-abc/route-maps/to-nat-gw/alarms/UUID, message=One or more relationships exist for object with id InternalRouteMapConfig/UUID2,errorId=PROVIDER_INVOCATION_FAILURE, path=null, apiError=error_code=3022, module_name=internal-framework, error_message='One or more relationships exist for object with id InternalRouteMapConfig/UUID3.', sourceSiteId=null].

• This issue is specific to upgrades from version 3.1.x to version 3.2.1 or later.

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

When a BGP neighbor is associated with a routeMap or prefix-list in version 3.1.x, and then the manager node is upgraded to version 3.2.1 or later, the BGP neighbor inadvertently forms a relationship with an unrelated route map or prefix list. As a result, attempts to delete the Tier0/RouteMap/PrefixList are unsuccessful as the system does not permit their deletion.

Resolution

Issue is fixed in NSX-T 3.2.2 version.

Workaround:
Step 1: Take the standard backup of the environment.

Step 2: Download the attached jar file and copy it to one of the manager nodes. (Any manager node would work but advice is to execute on the leader of the Corfu cluster.)
Verify the integrity of the jar file after copying to the NSX Manager with the following command.
Run this command from the place where the jar file is copied to.
MD5 (libcorfu-data-corrector-lib.jar) = fa0036bb4e3cbd2bac7784aa99427782

Step 3: Run the below command: 

java -cp "libcorfu-data-corrector-lib.jar:/opt/vmware/proton-tomcat/webapps/nsxapi/WEB-INF/lib/*:/usr/tomcat/lib/*"  -Dcorfu-property-file-path=/opt/vmware/proton-tomcat/conf/ufo-factory.properties -Djava.io.tmpdir=/image/corfu-tools/temp com.vmware.nsx.management.migration.impl.BgpNeighborDummyUpdate <username> <password>

Note: Enter Policy API credentials in place of <username> and <password> . The above script will execute the MP APIs to dummy update(without changing the payload) on each BgpNeighbor for each Tier0

Step 4: Delete the PrefixList/Routemap/Tier0 which currently stuck in deletion.