Pre-Upgrade check fails on one manager node due to incorrect upgrade

Products

VMware NSX

Issue/Introduction

Following error is seen for a newly added/ redeployed NSX Manager node during the NSX Manager prechecks:

UC is not updated on node ##############. Please upload the upgrade bundle and upgrade UC on one of these nodes to ensure the version of UC is same across all MP nodes.

Impact : Cannot proceed with Manager Upgrade
Below logs can be seen under /var/log/upgrade-coordinator/upgrade-coordinator.log

WARN pool-50-thread-6 UpgradeServiceImpl 914468 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] [PUC] Pre-upgrade check InspectionTaskInfo[id=ucVersionCheck,name=Check for the version sync status of UC on all the MP nodes.,description=This check ensures the version of UC is in sync across all the MP nodes before starting MP upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>] failed with result BasicInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[id=ucVersionCheck,name=Check for the version sync status of UC on all the MP nodes.,description=This check ensures the version of UC is in sync across all the MP nodes before starting MP upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>], failureMessages=null, failures=[{"moduleName":"upgrade-coordinator","errorCode":30486,"errorMessage":"UC is not updated on node ##############. Please upload the upgrade bundle and upgrade UC on one of these nodes to ensure the version of UC is same across all MP nodes."}]}

INFO pool-50-thread-6 UpgradeServiceImpl 914468 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [PUC] Finish Check for the version sync status of UC on all the MP nodes. on component MP. Result -- BasicInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[id=ucVersionCheck,name=Check for the version sync status of UC on all the MP nodes.,description=This check ensures the version of UC is in sync across all the MP nodes before starting MP upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>], failureMessages=null, failures=[{"moduleName":"upgrade-coordinator","errorCode":30486,"errorMessage":"UC is not updated on node ##############. Please upload the upgrade bundle and upgrade UC on one of these nodes to ensure the version of UC is same across all MP nodes."}]}

In the below example customer was trying to upgrade from current version of 4.2.0.0 to 4.2.2.1 and when checking the upgrade summary, Upgrade coordinator for two of the manager nodes got updated with target version (4.2.2.1) where as the problematic manager node UC is not updated with target version and it is still referring the current NSX version (4.2.0.0)
We also see the "upgrade_coordinator_updated": false for the problematic node
API to get the upgrade summary information of the Manager nodes:

GET https://<problematic nodeip>/api/v1/upgrade/summary --> Problematic manager node with incorrect version of UC

{
"system_version": "4.2.0.0.0.24105817",
"upgrade_coordinator_version": "4.2.0.0.0.24105821",
"upgrade_status": "PAUSED",
"target_version": "4.2.2.1.0.24765084",
"component_target_versions": [
{
"component_type": "EDGE",
"target_version": "4.2.2.1.0.24765090"
},
{
"component_type": "HOST",
"target_version": "4.2.2.1.0.24765085"
},
{
"component_type": "MP",
"target_version": "4.2.2.1.0.24765087"
}
],
"upgrade_coordinator_updated": false
}

GET https://<non problematic nodeip>/api/v1/upgrade/summary --> Non problematic Manager node which has the correct version of UC

{
"system_version": "4.2.0.0.0.24105817",
"upgrade_coordinator_version": "4.2.2.1.0.24765087",
"upgrade_status": "PAUSED",
"target_version": "4.2.2.1.0.24765084",
"component_target_versions": [
{
"component_type": "EDGE",
"target_version": "4.2.2.1.0.24765090"
},
{
"component_type": "HOST",
"target_version": "4.2.2.1.0.24765085"
},
{
"component_type": "MP",
"target_version": "4.2.2.1.0.24765087"
}
],
"upgrade_coordinator_updated": true
}

Environment

VMware NSX

Cause

The upgrade_coordinator_version did not get updated with the Target version and issue is seen as an error in the NSX Manager upgrade precheck

Resolution

Use the below POST API to update the UC version same across all MP nodes

https://<problematic node_ip>/api/v1/cluster/<node id of the problematic manager>/node/upgrade/uc

Body:
{
"product_version": "<target_version>" ---> The product version here needs to be updated with the "target_version" of the "component_type": "MP"
}

Sample API :

https://<problematic node_ip>/api/v1/cluster/<node id of the problematic manager>/node/upgrade/uc

Body:
{
"product_version": "4.2.2.1.0.24765084"
}

Below is the GET API output of the problematic node after updating the UC version with above POST API

GET https://<problematic node ip>/api/v1/upgrade/summary

{
"system_version": "4.2.0.0.0.24105817",
"upgrade_coordinator_version": "4.2.2.1.0.24765087",
"pre_upgrade_bundle_version": "4.2.2.1.0.24765084",
"upgrade_status": "PAUSED",
"target_version": "4.2.2.1.0.24765084",
"component_target_versions": [
{
"component_type": "EDGE",
"target_version": "4.2.2.1.0.24765090"
},
{
"component_type": "HOST",
"target_version": "4.2.2.1.0.24765085"
},
{
"component_type": "MP",
"target_version": "4.2.2.1.0.24765087"
}
],
"upgrade_coordinator_updated": true
}

Additional Information

"Check for the version sync status of UC on all the MP nodes" FAILURE seen for NSX precheck for upgrade

NSX upgrade pre-check fails with error "Pre-upgrade checks failed for MP: UC is not updated on node <NSX_MANAGER_Node(s)>. Please upgrade the UC on these nodes first and ensure the version of UC is same across all MP node"