NSX Manager upgrade fails during Data Migration steps with failure message "Unexpected error while upgrading upgrade unit: Data migration jar is not running"

Products

• VMware NSX

Issue/Introduction

• NSX is being upgraded to 4.0.x
• The upgrade of the NSX Manager fails during end stages of an upgrade that may have been launched by SDDC Manager (VCF) or directly initiated through NSX Manager. 
• NSX Manager updates are stuck at the Data Migration stage of Upgrade Steps after successfully completing the Node OS Upgrade.
• The NSX Manager nodes show that they are on the target version already as seen under System>Appliances.
• During one of several stages of the Data Migration Steps, the upgrade fails. Banner messages will state that the upgrade has failed and tell you to check error details to determine if manual resolution is needed and 'Retry Upgrade.'
• Upgrade Details show Failed to the right of any of the Upgrade Units. Clicking on the word 'Failed' will show the details which include the message:
  
  Unexpected error while upgrading upgrade unit: Data migration jar is not running.
  
  
• The below images shows examples of what is described above:
  
  
  

 

• The upgrade-coordinator.log will show entries like the below example stating that an error occurred while upgrading the given Upgrade Unit (in this example, it is showing a failure at TraceflowStatusMigrationTask):

2023-11-14T12:19:18.105Z ERROR task-executor-0-workitem-MP-TraceflowStatusMigrationTask WorkItem 3162 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30062" level="ERROR" subcomp="upgrade-coordinator"] Error encountered while upgrading upgrade unit TraceflowStatusMigrationTaskcom.vmware.nsx.management.upgrade.plugin.mp.exceptions.MPUpgradeException: null ... 2023-11-14T11:14:51.743Z  INFO task-executor-0-workitem-MP-TraceflowStatusMigrationTask MPRollingUpgradeServiceImpl 3162 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [MP UCP]  Triggering MP Upgrade for upgrade unit UpgradeUnit [id=TraceflowStatusMigrationTask, TransportNodeID =null, name=TraceflowStatusMigrationTask, description=null, type=MP, upgradeUnitSubtype=ACTION, currentVersion=Pending, warnings=null, errors=[], metaData={UU_TYPE=MIGRATION}, rebooting=false, UaReportedStatus=NOT_SYNCED, extendedConfiguration=null, progressTracker=UpgradeUnitProgressCollectorImpl [reference=TraceflowStatusMigrationTask, referenceType=MP, getProgressPercentage()=1, getUpgradeStatus()=IN_PROGRESS, getLastProgressMessage()=null, getParent()=ReferencedProgressCollectorImpl [reference=DataMigrationGroup, getProgressPercentage()=53, getUpgradeStatus()=IN_PROGRESS, getLastProgressMessage()=null]], class=class com.vmware.nsx.management.upgrade.model.UpgradeUnit, disabled=false,isSkipUpgrade=false, hashCode=-1765289747], plan null ... 2023-11-14T11:14:51.835Z  INFO task-executor-0-workitem-MP-TraceflowStatusMigrationTask MPRollingUpgradeServiceImpl 3162 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Data migration jar is not running

• NOTE: This issue applies to VMware NSX 4.0.x and below only. If you see similar issue on newer versions please see KB 405023
  
  
  
  

Environment

VMware NSX 4.0.x
VMware NSX-T Data Center 3.x

Cause

This is a known issue that can occur while upgrading between certain versions of NSX Manager. This issue occurs when a 2nd upgrade-coordinator upgrade of a different version of the desired upgrade is running.

Resolution

This issue is resolved in VMware NSX 4.1, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Workaround:

Attempt a Retry via the user interface.

Note: If a Retry fails, or for upgrades targeting versions 4.0.x or below that fail with the symptoms described here, please open a support case with Broadcom Support and refer to this KB article.