NSX-T Manager upgrade from 2.5.x to 3.x fails at run_migration_tool phase

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• NSX-T Data Center Manager upgrade from 2.5.x to 3.x
• An IP Block exists containing more than one IP Subnet each of size 1
• NSX-T Manager upgrade gets stuck at phase run_migration_tool on the orchestrator node

Manager01> get upgrade progress-status
****************************************************************************
Node Upgrade has been started. Please do not make any changes, until
the upgrade operation is complete. Run "get upgrade progress-status"
to show the progress of last upgrade step.
****************************************************************************
 
Upgrade info:
From-version: 2.5.2.1.0.16784099
To-version: 3.0.3.0.0.17777744
 
Upgrade steps:
download_os [2021-04-26 09:46:53 - 2021-04-26 09:47:23] SUCCESS
shutdown_manager [2021-04-26 09:47:32 - 2021-04-26 09:49:17] SUCCESS
install_os [2021-04-26 09:49:17 - 2021-04-26 09:50:26] SUCCESS
migrate_manager_config [2021-04-26 09:50:26 - 2021-04-26 09:50:31] SUCCESS
switch_os [2021-04-26 09:50:31 - 2021-04-26 09:50:36] SUCCESS
reboot [2021-04-26 09:50:36 - 2021-04-26 09:51:16] SUCCESS
run_migration_tool [2021-04-26 09:52:18 - ] IN_PROGRESS
    Status: Corfu Infrastructure Server is not running.

• NSX-T Manager /var/log/proton/data-migration.log may have logging similar to

2021-04-26T09:53:38.804Z ERROR main IpBlockSubnetAllocatorDeallocator - POOL-MGMT [nsx@6876 comp="nsx-manager" errorCode="MP5146" level="ERROR" subcomp="manager"] Insufficient Range in block IpBlock/<UUID> to allocate subnet of size 1 with requested start IP 10.20.30.1
2021-04-26T09:53:38.806Z ERROR main Migration - - [nsx@6876 comp="nsx-manager" errorCode="MP217" level="ERROR" subcomp="manager"]
com.vmware.nsx.management.common.exceptions.InsufficientResourcesException: null

Environment

VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center

Cause

During the upgrade, the NSX-T Manager proton service is restarted.
If there is an IP block with subnet of size 1, then the first subnet is processed correctly however any additional subnets within that IPBlock cannot be processed and an exception occurs blocking the upgrade.
This issue impacts upgrade from NSX-T Data Center 2.5.x to 3.x and does not affect 3.x to 3.x upgrades.

Resolution

This is a known issue affecting NSX-T Data Center, currently there is no resolution.

Workaround:
If you have experienced this upgrade failure, please open an SR with VMware Support .

To avoid this issue:
Prior to Manager upgrade, ensure that no IPBlock has more than 1 subnet of size 1.
To achieve this, it may be necessary to delete some IP subnets which can be re-added post upgrade.