NSX-T manager upgrade fails due to error while extracting upgrade bundle

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• You are upgrading VMware NSX.
• The Manager Upgrade Bundle (MUB) file extraction fails with the following error in the UI:

Upgrade-coordinator upgrade failed. Error - Error while extracting upgrade bundle VMware-NSX-upgrade-bundle-4.1.0.2.0.21761691.mub

• We see the following messages in the NSX-T Manager log: /var/log/upgrade-coordinator/upgrade-coordinator.log

2023-09-07T11:07:47.871Z INFO Thread-93 UpgradeBundleUcHelper 23098 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Failed to extract Upgrade Bundle VMware-NSX-upgrade->bundle-4.1.0.2.0.21761691.mub.
2023-09-07T11:07:18.040Z ERROR Thread-92 UpgradeCoordinatorUpgradeService 23098 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30040" level="ERROR" subcomp="upgrade-coordinator"] Error while updating upgrade-coordinator due to error Error while extracting upgrade bundle VMware-NSX-upgrade-bundle-4.1.0.2.0.21761691.mub : {1}.

• On the NSX-T Manager, In the log file /var/log syslog we see below entries which indicates that the UC upgrade failure is due to the creation of an extra file "migration-progress.json"
  2023-10-03T13:28:30.759Z MpPolicyController-manager2 - - - - [NSX] [nsx@6876 comp="nsx-manager" subcomp="upgrade-coordinator"] uc_helper: /image/upgrade-coordinator/ should contain upgrade bundle and eula file: 4102021761691.bundle_status, localized_eula, eula.txt, migration-progress.json, VMware-NSX-upgrade-bundle-4.1.0.2.0.21761691.tar.gz

Note: The preceding log excerpts are only examples. Date, time, versions and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center

Cause

After you successfully uploaded the MUB file, pre-checks were initiated.
Then the Upgrade Coordinator was upgraded, while the pre-checks are still ongoing.
Pre-check shouldn't be initiated until the Upgrade Coordinator is upgraded.

Resolution

In VMware NSX, Pre-check shouldn't be initiated until the Upgrade Coordinator is upgraded.

Workaround:
Due to incorrect procedure, pre checks triggered before UC is upgraded, a file is created in this location: /image/upgrade-coordinator/migration-progress.json

To workaround this issue, please proceed with the below steps on the orchestrator NSX Manager node:

• Delete the file that was created due to the pre-check being run after the MUB file was upload: 

rm /image/upgrade-coordinator/migration-progress.json

• Restart the upgrade coordinator to clean up stale mub-upload-in-progress flag file:

service upgrade-coordinator restart

• Then retry UC upgrade by clicking on "Begin Upgrade" button on Upgrade UI, and proceed with upgrade.