NSX manager pre-check fails with error "NSX Manager upgrade dry run failed. Do not proceed with the upgrade."

Products

VMware NSX

Issue/Introduction

NSX manager upgrade fails during pre-check with following error -

[nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] [PUC] Pre-upgrade check InspectionTaskInfo[id=upgradeEvaluationToolCheck,name=Upgrade Evaluation Tool execution check,description=Prompts user to execute Upgrade Evaluation Tool before proceeding with upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>] failed with result BasicInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[id=upgradeEvaluationToolCheck,name=Upgrade Evaluation Tool execution check,description=Prompts user to execute Upgrade Evaluation Tool before proceeding with upgrade.,componentType=MP,needsAcknowledgement=true,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>], failureMessages=null, failures=[{"moduleName":"upgrade-coordinator","errorCode":36000,"errorMessage":"NSX Manager upgrade dry run failed. Do not proceed with the upgrade. Please collect the support bundle and contact VMWare GS. Failed migrations: Starting parallel Corfu Exception during Manager dry-run : Traceback (most recent call last):
  File "/repository/4.2.1.0.0.24304122/Manager/dry-run/dry_run.py", line 276, in main
    start_parallel_corfu(dry_run_path)
  File "/repository/4.2.1.0.0.24304122/Manager/dry-run/dry_run.py", line 163, in start_parallel_corfu
    subprocess.check_output([str(fullcmd)],
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python3 /repository/4.2.1.0.0.24304122/Manager/dry-run/setup_parallel_corfu.py']' returned non-zero exit status 255.

Environment

VMware NSX

Cause

Upgrade coordinator in not upgraded to correct version on all nodes.

Resolution

Upgrade upgrade-coordinator to correct version.

1. Check upgrade coordinator version on all nodes using following API -

GET https://<nsx-manager>/api/v1/upgrade/summaryIf using Curl use the following command -curl -k -u admin -X GET https://<nsx-manager>/api/v1/upgrade/summary

Below are outputs from two manager nodes. Here upgrade coordinator in not updated on Manager -1.

Manager - 1

{
"system_version" : "4.1.0.2.0.21761691",
"upgrade_coordinator_version" : "4.1.0.2.0.21761695",
"pre_upgrade_bundle_version" : "4.1.0.2.0.21761691",
"upgrade_status" : "NOT_STARTED",
"target_version" : "4.2.1.0.0.24304122",
"upgrade_bundle_file_name" : "VMware-NSX-upgrade-bundle-4.2.1.0.0.24304122.mub",
"component_target_versions" : [ {
"component_type" : "MP",
"target_version" : "4.2.1.0.0.24302016"
} ],
"upgrade_coordinator_updated" : false

Manager - 2

{
"system_version" : "4.1.0.2.0.21761691",
"upgrade_coordinator_version" : "4.2.1.0.0.24302016",
"pre_upgrade_bundle_version" : "4.2.1.0.0.24304122",
"upgrade_status" : "NOT_STARTED",
"target_version" : "4.2.1.0.0.24304122",
"component_target_versions" : [ {
"component_type" : "MP",
"target_version" : "4.2.1.0.0.24302016"
} ],
"upgrade_coordinator_updated" : false
}

2. Update Upgrade coordinator using following API -

POST https://<nsx-manager>/api/v1/cluster/<node_uuid__of_affected_manager>/node/upgrade/uc Body: { "product_version": "<target_version>" }If using curl, use the following command. Here uc_upgrade.json contains body.curl -k -u admin -H "Content-Type:application/json" -X POST "https://<nsx-manager>/api/v1/cluster/<node_uuid__of_affected_manager>/node/upgrade/uc" -d @uc_upgrade.json

Additional Information

Pre-Upgrade check fails on one manager node due to incorrect upgrade_coordinator version