NSX Manager reports “Upgrade in progress” after restoring a backup taken after an upgrade is finished

Products

• VMware NSX

Issue/Introduction

• Complete upgrade to 4.2.x then restored NSX Manager after completing upgrade
  
  
• After performing an NSX Manager restore, the restore workflow completes, but when you log in to the NSX UI, you see an "An upgrade is currently in progress" notification.
  
   
  
  
• The upgrade appears to be left in a pending/paused state even after the restore finishes, and users cannot continue the normal upgrade flow from the UI.
  When opened "System" -> "Upgrade" in NSX Manager after restore, the "host upgrade" and "preparation for upgrade" is checked as completed, but "NSX Manager upgrade" is not checked.
  
  
• Looking at the /var/log/upgrade-coordinator/upgrade-coordinator.log on the NSX Manager, we see that the MP componentUpgradeStatus is paused at 70%.
  
  

  ####-##-##T##:##:##.###Z  INFO http-nio-127.0.0.1-7442-exec-1 UpgradeQueryServiceImpl ###### SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] For component type EDGE componentUpgradeStatus is SUCCESS 100%
  ####-##-##T##:##:##.###Z  INFO http-nio-127.0.0.1-7442-exec-1 UpgradeQueryServiceImpl ###### SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] For component type HOST componentUpgradeStatus is SUCCESS 100%
  ####-##-##T##:##:##.###Z  INFO http-nio-127.0.0.1-7442-exec-1 UpgradeQueryServiceImpl ###### SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] For component type MP componentUpgradeStatus is PAUSED 70% <-----!  

 

 

Environment

NSX 4.2.1.X
NSX 4.2.2.X

Cause

A migration control file used by the upgrade logic contained version-matching entries that were too broad and a formatting mistake.

As a result, certain migration tasks were incorrectly considered applicable after a restore of the same minor release build, so the UI continued to report an ongoing upgrade.

Resolution

This issue is resolved in VMware NSX 4.2.3, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround

If you are unable to upgrade to 4.2.3 immediately, you can manually clear the stalled upgrade state using the Upgrade Coordinator API.

Ensure the NSX Manager restore operation has completely finished.

Using a REST client or the command line, invoke the continue API against the NSX Manager:

Postman:

POST https://nsx-mgr-ip/api/v1/upgrade/plan?action=continue

CLI:

curl -k -u <username> -X POST "https://<nsx-mgr-ip>/api/v1/upgrade/plan?action=continue"

Navigate to System > Upgrade in the NSX UI and refresh your browser.

Additional Information

Note:

If the API request POST /api/v1/upgrade/plan?action=continue fails with the following error, the restoration is not yet complete.

{
  "httpStatus" : "BAD_REQUEST",
  "error_code" : 30955,
  "module_name" : "upgrade-coordinator",
  "error_message" : "Pre-upgrade checks failed for MP: Restore operation is in state: SUSPENDED_FOR_USER_ACTION. Please complete the restore operation before attempting to upgrade." <-----! 
}

Go to [System] > [Backup & Restore] and check if the restore status is "Suspended".

If the restore status is Suspended, select [Show Details], and carefully read the notes before selecting [RESUME] to resume the restore.