NSX-T Manager Upgrade fails on Orchestrator node step 7 of 13 "run_migration_tool"

Products

• VMware NSX

Issue/Introduction

Symptoms:
Orchestrator node fails to upgrade from 3.0.2 to 3.1.3.1

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 3.x

Cause

Transport nodes are deleted but VMs still exist as stale VM entries in NSX

Resolution

Fixed in version 3.2.0 from 3.0.x or 3.1.x

Workaround:
Warning: These steps require an additional script that is provided only with the guidance of VMware Support. Please open a support ticket before attempting them.

1. Set a maintenance window for the environment.
2. Check cluster is stable.
3. Take NSX-T backup of the Unified appliance.
4. Take a cold clone of all the unified appliances. Ensure the cloned machines are in powered-off state.
5. Stop all the services (Mainly proton, policy, ccp, proxy and CBM) leaving the corfu server running on all the managers.
6. Check if compactor is running, if yes, kill the compactor service. Compactor will not start again as CBM is down.
7. Now execute the cleanup script which can be obtained from VMware customer support

Additional Information

Before 3.0 these stale entries data were upgraded with the system and stale entries had no impact, but in 3.0 and further, we needed to upgrade data to fix this issue.

Impact/Risks:
Control plane is down and container VMs are unable to start