Unable to delete PrefixList/Tier0/RouteMap Post-Upgrade from 3.1.x to 3.2.X or higher

Products

• VMware NSX

Issue/Introduction

• After upgrading from NSX-T 3.1.x to 3.2.1 or later, you may be unable to delete Tier-0 Gateways, Route Maps, or Prefix Lists. The objects remain stuck in the "In Progress" state.
• The issue is identified by the error message "One or more relationships exist for object with id" when attempting to delete Tier0, Prefix List, or Route Map.
• In the UI, the object is greyed out and you are unable to make any changes.
• Logs similar to the below will be seen in the NSX Manager var/log/syslog

  Created alarm Alarm [policyPath=/infra/realized-state/enforcement-points/default/tier-0-logical-routers/edge01#########PROD/route-maps/AS-########/alarms/3456ad##-####-####-####-###0e70f##43a, 
  message=One or more relationships exist for object with id InternalRouteMapConfig/4fe113##-####-####-####-941d4f4daefc from 
  BgpNeighbor/123e85##-####-####-####-1##feb2b####, BgpNeighbor/53a45f##-####-####-####-5f4##f961####, 
  BgpNeighbor/8a888f##-####-####-####-11de3b2b####, ..., 
  BgpNeighbor/8#aef##-####-####-####-b4c44e6#d88e., 
  errorId=PROVIDER_INVOCATION_FAILURE, path=null, apiError=error_code=3022, module_name=internal-framework

• Logs similar to the below will also be under /var/log/proton/nsxapi.log.

  /var/log/proton/nsxapi.log:YYYY-MM-DDT09:45:03.538Z ERROR providerTaskExecutor-2 PolicyProviderUtil 19231 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM0" level="ERROR" subcomp="manager"] Created alarm Alarm [policyPath=/infra/realized-state/enforcement-points/default/tier-0-logical-routers/############/prefix-lists/acbouttztfd01/alarms/UUID, message=One or more relationships exist for object with id InternalPrefixListConfig/UUID1 from bgpNeighbor/UUID3,BgpNeighbor/UUID4
  /var/log/proton/nsxapi.log:YYYY-MM-DDT15:05:00.810Z hostname NSX 7447 POLICY [nsx@6876 comp="nsx-manager" errorCode="PM0" level="ERROR" subcomp="manager"] Created alarm Alarm [policyPath=/infra/realized-state/enforcement-points/default/tier-0-logical-routers/########/route-maps/to-nat-gw/alarms/UUID, message=One or more relationships exist for object with id InternalRouteMapConfig/UUID2,errorId=PROVIDER_INVOCATION_FAILURE, path=null, apiError=error_code=3022, module_name=internal-framework, error_message='One or more relationships exist for object with id InternalRouteMapConfig/UUID3.', sourceSiteId=null].

• This issue is specific to upgrades from version 3.1.x to version 3.2.1 or later.

Environment

VMware NSX-T Data Center 3.x

Cause

When a BGP neighbor is associated with a routeMap or prefix-list in version 3.1.x, and then the manager node is upgraded to version 3.2.1 or later, the BGP neighbor inadvertently forms a relationship with an unrelated route map or prefix list. As a result, attempts to delete the Tier0/RouteMap/PrefixList are unsuccessful as the system does not permit their deletion.

Resolution

As part of the upgrade, the issue will be fixed when upgrading to 3.2.4.

Workaround1:

• Step 1: Take the standard backup of the environment.
• Step 2: Download the attached jar file and copy it to one of the manager nodes. (Any manager node would work, but advice is to execute on the leader of the Corfu cluster.)
  Verify the integrity of the jar file after copying to the NSX Manager with the following command.
  Run this command from the place where the jar file is copied to.
  MD5 (libcorfu-data-corrector-lib.jar) = fa0036bb4e3cbd2bac7784aa99427782
  
   
• Step 3: Run the below command: 

  java -cp "libcorfu-data-corrector-lib.jar:/opt/vmware/proton-tomcat/webapps/nsxapi/WEB-INF/lib/*:/usr/tomcat/lib/*"  -Dcorfu-property-file-path=/opt/vmware/proton-tomcat/conf/ufo-factory.properties -Djava.io.tmpdir=/image/corfu-tools/temp com.vmware.nsx.management.migration.impl.BgpNeighborDummyUpdate <username> <password>
  

  Note: Enter Policy API credentials in place of <username> and <password> . The above script will execute the MP APIs to dummy update(without changing the payload) on each BgpNeighbor for each Tier0
• Step 4: Delete the PrefixList/Routemap/Tier0, which is currently stuck in deletion.

Workaround2:

• Step 1: Take the standard backup of the environment.
• Step 2: Download the attached nsx_bgp_script_main_new.py.zip and upload it to the NSX Manager.
• Step 3: Log in to the NSX manager as root via SSH. 

  unzip nsx_bgp_script.zip 

• Step 4: Access the unzipped directory.

  cd nsx_bgp_script

• Step 5: Run the script. 

  python nsx_bgp_script_main.py