NSX-T MP Upgrade failed on step 7 (run_migration_tool) on LM on federated environment
search cancel

NSX-T MP Upgrade failed on step 7 (run_migration_tool) on LM on federated environment

book

Article ID: 330531

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

Symptoms:
  • MP upgrade stuck after failing on step 7 (run_migration_tool) .

dr-lm1> get upgrade progress-status
****************************************************************************
Node Upgrade has been started. Please do not make any changes, until
the upgrade operation is complete. Run "get upgrade progress-status"
to show the progress of last upgrade step.
****************************************************************************
Tue Aug 02 2022 UTC 14:37:46.753
Upgrade info:
From-version: 3.2.0.0.0.19067081
To-version: 3.2.1.1.0.20115694

​Upgrade steps:
download_os [2022-07-28 14:51:09 - 2022-07-28 14:51:39] SUCCESS
shutdown_manager [2022-07-28 14:51:47 - 2022-07-28 14:53:32] SUCCESS
install_os [2022-07-28 14:53:32 - 2022-07-28 14:54:39] SUCCESS
migrate_manager_config [2022-07-28 14:54:39 - 2022-07-28 14:54:45] SUCCESS
switch_os [2022-07-28 14:54:45 - 2022-07-28 14:54:54] SUCCESS
reboot [2022-07-28 14:54:54 - 2022-07-28 18:55:29] SUCCESS
run_migration_tool [2022-07-28 18:57:53 - 2022-07-28 18:58:28] FAILED
run_migration_tool [2022-07-28 19:21:30 - 2022-07-28 19:22:05] FAILED
run_migration_tool [2022-07-28 19:26:55 - 2022-07-28 19:27:30] FAILED
run_migration_tool [2022-07-28 19:49:20 - 2022-07-28 19:49:55] FAILED
run_migration_tool [2022-07-28 19:50:29 - 2022-07-28 19:51:04] FAILED
pin_ui [2022-07-28 19:52:19 - 2022-07-28 19:52:24] SUCCESS
start_manager [2022-07-28 20:57:00 - ] FAILED
run_migration_tool [2022-07-28 21:23:48 - 2022-07-28 21:24:28] FAILED
----- Output of last step start ------
Status:
2022-07-28 21:23:51.409811 Deleting datastore files
2022-07-28 21:23:51.423670 Copying old datastore files
2022-07-28 21:23:51.561819 Done copying old datastore files
2022-07-28 21:23:53.461452 Start Corfu server
2022-07-28 21:23:58.041919 Process corfu-server started
2022-07-28 21:24:25.390044 Error running logical data migration tool. return value 1, log file /var/log/proton/logical-migration.log

​Overall Progress: (0/6)
---- (1) CCP: Pending
--------------------------------------------------------------------------------------------

---- (2) Proton: Pending

--------------------------------------------------------------------------------------------

---- (3) Policy: Pending

--------------------------------------------------------------------------------------------

---- (4) Logical: 0% [2 of 8637 object(s)] (2022-07-28 05:24:23 - ) ----

Currently Migrating: RoutingConfigPathToUuidMigrationTask 0% [0 of 66 objects] (2022-07-28 05:24:24 - )

--------------------------------------------------------------------------------------------

---- (5) CBM: Pending

--------------------------------------------------------------------------------------------

---- (6) UFO Checkpointing: Pending

--------------------------------------------------------------------------------------------
Stdout: Starting Manager run_migration_tool script Ending Manager run_migration_tool script ​Troubleshooting: Upgrade has failed, and retry may not work. Appliance OS is of a new version, however, UI will not be available. Please contact GSS to rollback the system to the previous version.
------ Output of last step end ------

  • In the /var/log/proton/logical-migration.log file, you see entries similar to:

2022-07-28T18:58:27.835Z  WARN main UfoCorfuTableMigrator 3773 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="manager"] ERROR while running logical migration MappingDetails{modelName='null', migrationType=null, reason='This migration task will run for migrations from GC/HL/3.2.0 to 3.2.1. It will convert path id into uuid as id.', customMigratorClassName='com.vmware.nsx.management.migration.impl.RoutingConfigPathToUuidMigrationTask', fieldMappings=null, targetProtoName='null', requiresCustomCode='false', owner='null', apiToTest='null'}
java.lang.NullPointerException: null
        at com.vmware.nsx.management.migration.impl.RoutingConfigPathToUuidMigrationTask.setInternalPrefixListMatch(RoutingConfigPathToUuidMigrationTask.java:510) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.impl.RoutingConfigPathToUuidMigrationTask.prepareRouteMapInternalConfig(RoutingConfigPathToUuidMigrationTask.java:435) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.impl.RoutingConfigPathToUuidMigrationTask.migrateRouteMap(RoutingConfigPathToUuidMigrationTask.java:412) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.impl.RoutingConfigPathToUuidMigrationTask.migrate(RoutingConfigPathToUuidMigrationTask.java:170) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.ufo.UfoCorfuTableMigrator.migrate(UfoCorfuTableMigrator.java:138) [logical-migration.jar:?]
        at com.vmware.nsx.management.migration.ufo.UFOMigration.migrate(UFOMigration.java:235) [logical-migration.jar:?]
        at com.vmware.nsx.management.migration.impl.LogicalMigration.executeMigrations(LogicalMigration.java:43) [logical-migration.jar:?]
        at com.vmware.nsx.management.migration.impl.Migration.migrate(Migration.java:46) [logical-migration.jar:?]
        at com.vmware.nsx.management.migration.impl.LogicalMigration.main(LogicalMigration.java:29) [logical-migration.jar:?]
2022-07-28T18:58:27.845Z ERROR main Migration 3773 - [nsx@6876 comp="nsx-manager" errorCode="MP217" level="ERROR" subcomp="manager"] Migration failed
com.vmware.nsx.management.common.exceptions.BaseException: ERROR while running logical migration MappingDetails{modelName='null', migrationType=null, reason='This migration task will run for migrations from GC/HL/3.2.0 to 3.2.1. It will convert path id into uuid as id.', customMigratorClassName='com.vmware.nsx.management.migration.impl.RoutingConfigPathToUuidMigrationTask', fieldMappings=null, targetProtoName='null', requiresCustomCode='false', owner='null', apiToTest='null'}
        at com.vmware.nsx.management.migration.ufo.UfoCorfuTableMigrator.migrate(UfoCorfuTableMigrator.java:142) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.ufo.UFOMigration.migrate(UFOMigration.java:235) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.impl.LogicalMigration.executeMigrations(LogicalMigration.java:43) ~[logical-migration.jar:?]
        at com.vmware.nsx.management.migration.impl.Migration.migrate(Migration.java:46) [logical-migration.jar:?]
        at com.vmware.nsx.management.migration.impl.LogicalMigration.main(LogicalMigration.java:29) [logical-migration.jar:?]
Caused by: java.lang.NullPointerException
      ...TRUNCATED

  • You may encounter the same while running Evaluation Tool --> KB -NSX Upgrade Evaluation Tool (87379).

nsx-upgrade> start dry-run data-migration mp-ip 10.10.XX.XX
Root password of the Remote MP node:
(1/11) Checking ssh connectivity to the MP node 10.10.XX.XX with root user...
(2/11) Creating a temporary folder on MP 10.10.XX.XX
(3/11) Copy Corfu data to the temporary folder on MP 10.10.XX.XX
(4/11) Copy nsx_issue file to the temporary folder on MP 10.10.XX.XX
(5/11) Create tar of the temporary folder on MP 10.10.X.XX
(6/11) Delete the temporary folder on MP 10.10.XX.XX
(7/11) Fetching tar containing Corfu data
(8/11) Delete tar file on MP
(9/11) Downloaded corfu tgz file of size 35 MB
(10/11) Loading the fetched Corfu data
(11/11) Starting data-migration dry-run
Running.... Please track progress in /var/log/cloudnet/data-migration.log, /var/log/proton/data-migration.log, /var/log/policy/data-migration.log, /var/log/proton/logical-migration.log
*** WARNING: Some pre-upgrade check(s) failed. Do not proceed with the upgrade. Please collect the support bundle and contact VMWare GSS***RoutingConfigPathToUuidMigrationTask: java.lang.NullPointerException

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

 


Environment

VMware NSX-T Data Center

Cause

Global-T0 has been deleted and also the system-created prefix lists. But there are still stale system-created route map and community-list entries for Global-T0 (for federation use). The stale system-created route maps are still referring to the deleted prefix lists. This causes migration failure.

Resolution

Currently there is no resolution to this issue.

 


Workaround:
  • We need to delete the community and prefix stale entries for the deleted T0 Gateway within Corfu database.
  • If you believe you have encountered this issue, please open a support request and refer to this KB article.

 

 

 

 

 


Powered by Wolken Software