NSX-T manager upgrade fails due to error while extracting upgrade bundle

Products

• VMware NSX

Issue/Introduction

• You are upgrading VMware NSX.
• The Manager Upgrade Bundle (MUB) file extraction fails with the following error in the UI:
  Upgrade-coordinator upgrade failed. Error - Error while extracting upgrade bundle VMware-NSX-upgrade-bundle-4.1.0.2.0.21761691.mub

• We see the following messages in the NSX-T Manager log: /var/log/upgrade-coordinator/upgrade-coordinator.log
  2023-09-07T11:07:47.871Z INFO Thread-93 UpgradeBundleUcHelper 23098 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Failed to extract Upgrade Bundle VMware-NSX-upgrade->bundle-4.1.0.2.0.21761691.mub.
2023-09-07T11:07:18.040Z ERROR Thread-92 UpgradeCoordinatorUpgradeService 23098 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30040" level="ERROR" subcomp="upgrade-coordinator"] Error while updating upgrade-coordinator due to error Error while extracting upgrade bundle VMware-NSX-upgrade-bundle-4.1.0.2.0.21761691.mub : {1}.

• On the NSX-T Manager, In the log file /var/log syslog we see below entries which indicates that the UC upgrade failure is due to the creation of an extra file "migration-progress.json"
  2023-10-03T13:28:30.759Z MpPolicyController-manager2 - - - - [NSX] [nsx@6876 comp="nsx-manager" subcomp="upgrade-coordinator"] uc_helper: /image/upgrade-coordinator/ should contain upgrade bundle and eula file: 4102021761691.bundle_status, localized_eula, eula.txt, migration-progress.json, VMware-NSX-upgrade-bundle-4.1.0.2.0.21761691.tar.gz

Note: The preceding log excerpts are only examples. Date, time, versions and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center

VMware NSX

Cause

• Pre-checks were initiated manually right after the MUB file is successfully uploaded. UC will be busy extracting and verifying the uploaded bundle.
• Due to incorrect procedure, the upgrade workflow will fail and a file may get created in this location: /image/upgrade-coordinator/migration-progress.json
  

Resolution

Do not interrupt the default bundle extraction or UC upgrade progress after the MUB is uploaded. Continue to next step or run pre-checks only when the MUB file upload and subsequent default process is complete. 

Workaround:
To workaround this issue, please proceed with the below steps on the orchestrator NSX Manager node:

• Delete the file that was created due to the pre-check being run after the MUB file was upload:
  rm /image/upgrade-coordinator/migration-progress.json

• Restart the upgrade coordinator to clean up stale mub-upload-in-progress flag file:
  service upgrade-coordinator restart

• Then retry UC upgrade by clicking on "Begin Upgrade" button on Upgrade UI, and proceed with upgrade.