NSX manager pre-check fails with error "NSX Manager upgrade dry run failed. Do not proceed with the upgrade."
search cancel

NSX manager pre-check fails with error "NSX Manager upgrade dry run failed. Do not proceed with the upgrade."

book

Article ID: 412826

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

NSX manager upgrade fails during pre-check with following error -

[nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] [PUC] Pre-upgrade check InspectionTaskInfo[id=upgradeEvaluationToolCheck,name=Upgrade Evaluation Tool execution check,description=Prompts user to execute Upgrade Evaluation Tool before proceeding with upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>] failed with result BasicInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[id=upgradeEvaluationToolCheck,name=Upgrade Evaluation Tool execution check,description=Prompts user to execute Upgrade Evaluation Tool before proceeding with upgrade.,componentType=MP,needsAcknowledgement=true,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>], failureMessages=null, failures=[{"moduleName":"upgrade-coordinator","errorCode":36000,"errorMessage":"NSX Manager upgrade dry run failed. Do not proceed with the upgrade. Please collect the support bundle and contact VMWare GS. Failed migrations: Starting parallel Corfu Exception during Manager dry-run : Traceback (most recent call last):
  File "/repository/4.2.1.0.0.24304122/Manager/dry-run/dry_run.py", line 276, in main
    start_parallel_corfu(dry_run_path)
  File "/repository/4.2.1.0.0.24304122/Manager/dry-run/dry_run.py", line 163, in start_parallel_corfu
    subprocess.check_output([str(fullcmd)],
  File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['python3 /repository/4.2.1.0.0.24304122/Manager/dry-run/setup_parallel_corfu.py']' returned non-zero exit status 255.

Environment

VMware NSX

Cause

  • Upgrade coordinator in not upgraded to correct version on all nodes.
  • If UC version has been verified to be consistent across all managers, this error can also occur if the Datastore cluster is in a degraded state.

Resolution

Update upgrade-coordinator to correct version:

  • Check upgrade coordinator version on all nodes using following API:
    • GET https://<nsx-manager>/api/v1/upgrade/summary
    • If using Curl use the following command -
      curl -k -u admin -X GET https://<nsx-manager>/api/v1/upgrade/summary

Below are outputs from two manager nodes. Here upgrade coordinator in not updated on Manager -1.

Manager - 1

{
  "system_version" : "4.1.0.2.0.21761691",
  "upgrade_coordinator_version" : "4.1.0.2.0.21761695",
  "pre_upgrade_bundle_version" : "4.1.0.2.0.21761691",
  "upgrade_status" : "NOT_STARTED",
  "target_version" : "4.2.1.0.0.24304122",
  "upgrade_bundle_file_name" : "VMware-NSX-upgrade-bundle-4.2.1.0.0.24304122.mub",
  "component_target_versions" : [ {
    "component_type" : "MP",
    "target_version" : "4.2.1.0.0.24302016"
  } ],
  "upgrade_coordinator_updated" : false

Manager - 2

{
  "system_version" : "4.1.0.2.0.21761691",
  "upgrade_coordinator_version" : "4.2.1.0.0.24302016",
  "pre_upgrade_bundle_version" : "4.2.1.0.0.24304122",
  "upgrade_status" : "NOT_STARTED",
  "target_version" : "4.2.1.0.0.24304122",
  "component_target_versions" : [ {
    "component_type" : "MP",
    "target_version" : "4.2.1.0.0.24302016"
  } ],
  "upgrade_coordinator_updated" : true
}

  • Update the upgrade coordinator version for incorrect entry using following API:
    • POST https://<nsx-manager>/api/v1/cluster/<node_uuid__of_affected_manager>/node/upgrade/uc
      Body:
      {
        "product_version": "<target_version>"  
      }
    • If using curl, use the following command. Here uc_upgrade.json contains body.
      curl -k -u admin  -H "Content-Type:application/json" -X POST "https://<nsx-manager>/api/v1/cluster/<node_uuid__of_affected_manager>/node/upgrade/uc" -d @uc_upgrade.json

 

Verify NSX Manager Cluster status to be stable for all services by running get cluster status command in NSX admin CLI. If datastore service is found to be down on any manager node, refer to below KBs for further troubleshooting:

Additional Information

Powered by Wolken Software