Unexpected error while upgrading upgrade unit: Data migration jar is not running error observed upgrade of NSX in step Data Migration

Products

• VMware NSX

Issue/Introduction

• NSX Manager upgrade fails during Data Migration step with error "Unexpected error while upgrading upgrade unit: Data migration jar is not running"
• Data migration jar is not running on the NSX manager nodes.
  /var/log/upgrade-coordinator/upgrade-coordinator.logs
upgrade-coordinator.log:33528:##-##-##T06:16:15.503Z INFO task-executor-5-1-workitem-MP-IdsProfileMigrationTask MPRollingUpgradeServiceImpl 75317 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Data migration jar is not running
• Repository is missing new version's folder.
• Repository content on orchestrator:

drwxr-xr-x 26 root root 4.0K Oct 7 16:53 ..
drwxrwx--- 6 uuc grepodir 4.0K Sep 30 15:59 .
drwxrwx--- 7 uuc grepodir 4.0K Sep 30 14:09 4.2.1.4.0.24765227
drwxrwx--- 6 uuc grepodir 4.0K Jun 2 20:25 4.2.1.4.0.24771817
-rwxrw---- 1 uuc grepodir 18 Jun 7 2023 current_version
drwxrwx--- 7 uuc grepodir 4.0K May 13 2023 4.1.0.2.0.21761691
  
  4.2.1.4.0.24765227 is missing on node where upgrade-coordinator is running -
  
  drwxr-xr-x 28 root root 4.0K Oct 7 16:50 ..
drwxrwx--- 4 uuc grepodir 4.0K Oct 6 09:02 .
drwxrwx--- 6 uuc grepodir 4.0K Sep 30 16:05 4.2.1.4.0.24771817
-rwxrw---- 1 uuc grepodir 18 Jun 7 2023 current_version
drwxrwx--- 2 uuc grepodir 16K Oct 4 2021 lost+found

Environment

VMware NSX 4.1.0.2

Cause

• MP fabric module was updated multiple times via API. These updates inadvertently removed the 'from-version' and 'to-version' MUB repo-list entries, which consequently caused the deletion of the corresponding directories from the other two nodes. 
• Repo Sync was incorrectly marked as successful across all three nodes.

Resolution

Both "from" and "to" version directories must be present for upgrade.

1. Restore the NSX manager with the backup taken before upgrade process and continue the upgrade process again
2. Collect the output of the same API and validate that the 'repo-sync' is successful on all three nodes
   GET https://<MP_IP>/api/v1/fabric/modules
3. Please verify that the '/repository' partition contains both the "from" and "to" version directories.

Additional Information

• For back and restore, refer to Backup and Restore During Upgrade.
• If the issue is same but the symptoms aren't matching, refer to KB article NSX Manager upgrade fails during Data Migration steps with failure message "Unexpected error while upgrading upgrade unit: Data migration jar is not running"