Upgrading NSX Global Managers fails with "Incorrect bundle of version..."

Products

VMware NSX

Issue/Introduction

After the upgrade precheck and preparation of nodes is successful, the Global Manager OS upgrade fails with the error:

"Unexpected error while upgrading upgrade unit [MPP] Node upgrade failed: Incorrect bundle of version 4.... found on node".

Further log messages found in the /var/log/upgrade-coordinator/upgrade-coordinator.log will be similar to the output below (please note the version of the bundle may differ based on the upgrade version and the logs below are not an exhaustive list of log snippets related to this issue):

INFO http-nio-127.0.0.1-7442-exec-6 UpgradeQueryServiceImpl 2976 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] For component type MP componentUpgradeStatus is SUCCESS 0%

INFO http-nio-127.0.0.1-7442-exec-6 UpgradeCoordinatorFacadeImpl 2976 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Got upgrade status summary overallUpgradeStatus: SUCCESS

INFO http-nio-127.0.0.1-7442-exec-5 UpgradeServiceImpl 2629378 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Triggering upgrade of MP component

INFO task-executor-19-1-workitem-MP-########-####-####-####-############# WorkItem 2629378 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Calling upgrade for upgrade unit: Manager_Name_Here (########-####-####-####-#############)

INFO task-executor-19-1-workitem-MP-########-####-####-####-############# MPRollingUpgradeServiceImpl 2629378 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Node ########-####-####-####-############# in already in maintenance mode

INFO task-executor-19-1-workitem-MP-########-####-####-####-############# MPRollingUpgradeServiceImpl 2629378 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [MP UCP] Upgrade bundle of version 4.1.2.6.0.24723857 found on node.

 INFO task-executor-19-1-workitem-MP-########-####-####-####-############# ExecutionMonitorServiceImpl 2629378 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Execution monitor service invoked to react to failure of node ########-####-####-####-############# [[MPP] Node upgrade failed : Incorrect bundle of version 4.1.2.6.0.24723857 found on node..]

Environment

VMware NSX 4.x

Cause

The management plane upgrade has failed because a previously extracted bundle was present in the /image/ partition.

Resolution

Open an SSH session into the Global Manager that is reporting the failure with admin credentials
Enable debug mode by running the command set debug-mode
Run the following command to remove the old bundle: start upgrade-bundle VMware-NSX-unified-appliance-<bundle version here> step finish_upgrade
- Example command: start upgrade-bundle VMware-NSX-unified-appliance-4.1.2.6.0.24723857 step finish_upgrade
- The output should look similar to the below example once completed:
  - Confirmation of process being completed can be found by running the command upgrade progress-status and the results reporting "Upgrade is not in progress".
Log into the NSX VIP UI and retry the NSX Manager upgrade.

If the above steps do not resolve the issue, please open a support case with Broadcom Support and reference this KB.

Please reference the below document for assistance with opening a support case.

Creating and managing Broadcom support cases

Please ensure that the NSX Manager log bundles have been gathered and uploaded to the support case for review. Please refer to the below document for assistance with gathering logs and uploading them to a VMware by Broadcom support case.