"UC is not updated on node" NSX Manager upgrade pre-check error

Products

VMware NSX

Issue/Introduction

On the NSX UI, System > Upgrade, the Manager pre-check fails with an upgrade blocking error

UC is not updated on node ##############. Please upload the upgrade bundle and upgrade UC on one of these nodes to ensure the version of UC is same across all MP nodes.

The NSX Manager upgrade log, /var/log/upgrade-coordinator/upgrade-coordinator.log, might have a log message similar to this example

WARN pool-50-thread-6 UpgradeServiceImpl 914468 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] [PUC] Pre-upgrade check InspectionTaskInfo[id=ucVersionCheck,name=Check for the version sync status of UC on all the MP nodes.,description=This check ensures the version of UC is in sync across all the MP nodes before starting MP upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>] failed with result BasicInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[id=ucVersionCheck,name=Check for the version sync status of UC on all the MP nodes.,description=This check ensures the version of UC is in sync across all the MP nodes before starting MP upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>], failureMessages=null, failures=[{"moduleName":"upgrade-coordinator","errorCode":30486,"errorMessage":"UC is not updated on node <Manager node>. Please upload the upgrade bundle and upgrade UC on one of these nodes to ensure the version of UC is same across all MP nodes."}]}

INFO pool-50-thread-6 UpgradeServiceImpl 914468 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [PUC] Finish Check for the version sync status of UC on all the MP nodes. on component MP. Result -- BasicInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[id=ucVersionCheck,name=Check for the version sync status of UC on all the MP nodes.,description=This check ensures the version of UC is in sync across all the MP nodes before starting MP upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>], failureMessages=null, failures=[{"moduleName":"upgrade-coordinator","errorCode":30486,"errorMessage":"UC is not updated on node <Manager node>. Please upload the upgrade bundle and upgrade UC on one of these nodes to ensure the version of UC is same across all MP nodes."}]}

When checking the upgrade summary, Upgrade Coordinator(UC) for two of the manager nodes got upgraded to the target version however one node has not had its UC upgraded.
The Upgrade Coordinator(UC) version is confirmed via API, in this example NSX is being upgraded from 4.2.0.0 to 4.2.2.1. The "upgrade_status" will either be "PAUSED" or "NOT STARTED".

GET https://<problematic manager node>/api/v1/upgrade/summary --> Problem manager node

{
"system_version": "4.2.0.0.0.24105817",
"upgrade_coordinator_version": "4.2.0.0.0.24105821", <<< UC version has not been upgraded to the target version
"upgrade_status": "PAUSED",
"target_version": "4.2.2.1.0.24765084",
"component_target_versions": [
{
"component_type": "EDGE",
"target_version": "4.2.2.1.0.24765090"
},
{
"component_type": "HOST",
"target_version": "4.2.2.1.0.24765085"
},
{
"component_type": "MP",
"target_version": "4.2.2.1.0.24765087"
}
],
"upgrade_coordinator_updated": false <<<< UC has not been upgraded
}

GET https://<non problematic Manager node>/api/v1/upgrade/summary --> Good manager node which has the correct version of UC

{
"system_version": "4.2.0.0.0.24105817",
"upgrade_coordinator_version": "4.2.2.1.0.24765084", <<<< UC matches the target version
"upgrade_status": "PAUSED",
"target_version": "4.2.2.1.0.24765084",
"component_target_versions": [
{
"component_type": "EDGE",
"target_version": "4.2.2.1.0.24765090"
},
{
"component_type": "HOST",
"target_version": "4.2.2.1.0.24765085"
},
{
"component_type": "MP",
"target_version": "4.2.2.1.0.24765087"
}
],
"upgrade_coordinator_updated": true <<<< UC has been upgraded
}

Environment

VMware NSX

Cause

The upgrade_coordinator_version did not get updated with the Target version and issue is seen as an error in the NSX Manager upgrade precheck.

Resolution

This is a known issue impacting VMware NSX.

To resolve the issue:

Run the GET command against the problem node

GET https://<problematic manager node>/api/v1/upgrade/summary

{
"system_version": "4.2.0.0.0.24105817",
"upgrade_coordinator_version": "4.2.0.0.0.24105821",
"upgrade_status": "PAUSED",
"target_version": "4.2.2.1.0.24765084", <<<<< Use this value in step 2
"component_target_versions": [
{
"component_type": "EDGE",
"target_version": "4.2.2.1.0.24765090"
},
{
"component_type": "HOST",
"target_version": "4.2.2.1.0.24765085"
},
{
"component_type": "MP",
"target_version": "4.2.2.1.0.24765087" <<<< Do not use this value in step 2
}
],
"upgrade_coordinator_updated": false
}
Identify the UUID of the problem NSX Manager

ssh as admin to any NSX Manager run the command get node, copy the UUID of the problem Manager
Use the below POST API to update the UC version on the problem node. If Postman is available then running the POST command with the POST application will be the easiest option.
POST https://<problematic node_ip>/api/v1/cluster/<UUID of Manager from step 2>/node/upgrade/uc

Body:
{
"product_version": "4.2.2.1.0.24765084" <<< Replace current value with the target version from step 1
}

If using the curl command instead then the command will be in the following format and you will need to pass a json file with the text above. Please note it is needed to include the '@' symbol in front of the filename when running the command. An example json file is attached to the KB.
curl -k -u admin -H "Content-Type:application/json" -X POST "https://<nsx-manager>/api/v1/cluster/<node_uuid__of_affected_manager>/node/upgrade/uc" -d @uc_upgrade.json
Confirm the change has taken effect

GET https://<problematic manager node>/api/v1/upgrade/summary

{
"system_version": "4.2.0.0.0.24105817",
"upgrade_coordinator_version": "4.2.2.1.0.24765084", <<<< UC version now matching target version
"pre_upgrade_bundle_version": "4.2.2.1.0.24765084",
"upgrade_status": "PAUSED",
"target_version": "4.2.2.1.0.24765084",
"component_target_versions": [
{
"component_type": "EDGE",
"target_version": "4.2.2.1.0.24765090"
},
{
"component_type": "HOST",
"target_version": "4.2.2.1.0.24765085"
},
{
"component_type": "MP",
"target_version": "4.2.2.1.0.24765087"
}
],
"upgrade_coordinator_updated": true
}

Additional Information

"Check for the version sync status of UC on all the MP nodes" FAILURE seen for NSX precheck for upgrade

NSX upgrade pre-check fails with error "Pre-upgrade checks failed for MP: UC is not updated on node <NSX_MANAGER_Node(s)>. Please upgrade the UC on these nodes first and ensure the version of UC is same across all MP node"

Attachments

uc_upgrade.json get_app