NSX Manager upgrade stuck "In Progress... 67%"

Products

• VMware NSX

Issue/Introduction

 

• Upgrade coordinator shows that the NSX upgrade is in progress:
  
  
  
• The progress of NSX Manager upgrade shows that it is "In Progress... 67%" but one of NSX Manager upgrade has not started:
  
  
  
• Checking the upgrade progress status shows that the"Upgrade is not in progress":
  
  • Login to NSX Manager CLI using admin account and run the following command:
NSX-manager> get upgrade progress-status
<Timestamp>
Upgrade is not in progress
    
    
•  Checking NSX version using NSX CLI command "get version", it shows that  NSX version is already on the upgraded version. (Confirm the same on all three NSX Managers)
• The following messages are found in /var/log/upgrade-coordinator/upgrade-coordinator.log:
  <Timestamp>  INFO WrapperStartStopAppMain UpgradeServiceImpl 3339 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Bundle is not uploaded yet. Target version is null for component MP. Hence skipping the overall upgrade status update.
  
  
• After performing the Reset NSX Upgrade Prechecks, the status of the upgrade still stays as "In Progress" 

Environment

VMware NSX 4.x

Cause

As per error message found in the upgrade-coordinator logs, the upgrade bundle seems to be removed from NSX Manager.

Resolution

The following workaround can be used:

1. Reset NSX Upgrade Plan for MP using the following API call (as per How to Reset NSX Upgrade Prechecks): 
   
   

   NOTE:

   Check to make sure the Upgrade-Coordinator service is running before and after running the above API calls. 

   
   
   POST https://<mgr_IP>/api/v1/upgrade/plan?action=reset&component_type=MP
   
   
2. Upload .PUB file to upgrade coordinator:
   1. Login to NSX UI
   2.  go to System > Upgrade 
   3. Click on Upgrade
   4. Upload .pub file (Make sure that correct destination version is downloaded)
   5. upload the latest version of .pub file and run the pre-upgrade check again.
   6. continue to run  pre-checks until upgrade coordinator confirms that the upgrade has been successful.

Additional Information

Upgrade the Upgrade Coordinator

Upgrade Order

Starting

NSX

4.0.1.1, you have the flexibility to change the order of upgrade for your edge clusters and hosts. You can alternate between groups of hosts and groups of edge nodes during upgrade.

For instance, if you have two edge clusters and two host clusters, you can first upgrade one of the edge clusters followed by a host cluster and then complete upgrade for the remaining clusters in any order you prefer.

You can either upgrade a cluster of hosts at a time or a cluster of edges.
Host and edge nodes cannot be upgraded in parallel. The

NSX Manager

is upgraded only after all the edges and hosts have been upgraded.