"UC is not updated on node" NSX Manager upgrade pre-check error

Products

• VMware NSX

Issue/Introduction

• On the NSX UI, System > Upgrade, the Manager pre-check fails with an upgrade blocking error 

UC is not updated on node ##############. Please upload the upgrade bundle and upgrade UC on one of these nodes to ensure the version of UC is same across all MP nodes.

• The NSX Manager upgrade log, /var/log/upgrade-coordinator/upgrade-coordinator.log, has a log message similar to this example

WARN pool-50-thread-6 UpgradeServiceImpl 914468 SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] [PUC] Pre-upgrade check InspectionTaskInfo[id=ucVersionCheck,name=Check for the version sync status of UC on all the MP nodes.,description=This check ensures the version of UC is in sync across all the MP nodes before starting MP upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>] failed with result BasicInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[id=ucVersionCheck,name=Check for the version sync status of UC on all the MP nodes.,description=This check ensures the version of UC is in sync across all the MP nodes before starting MP upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>], failureMessages=null, failures=[{"moduleName":"upgrade-coordinator","errorCode":30486,"errorMessage":"UC is not updated on node <Manager node>. Please upload the upgrade bundle and upgrade UC on one of these nodes to ensure the version of UC is same across all MP nodes."}]}

INFO pool-50-thread-6 UpgradeServiceImpl 914468 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [PUC] Finish Check for the version sync status of UC on all the MP nodes. on component MP. Result -- BasicInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[id=ucVersionCheck,name=Check for the version sync status of UC on all the MP nodes.,description=This check ensures the version of UC is in sync across all the MP nodes before starting MP upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>], failureMessages=null, failures=[{"moduleName":"upgrade-coordinator","errorCode":30486,"errorMessage":"UC is not updated on node <Manager node>. Please upload the upgrade bundle and upgrade UC on one of these nodes to ensure the version of UC is same across all MP nodes."}]}

• When checking the upgrade summary, Upgrade Coordinator(UC) for two of the manager nodes got upgraded to the target version however one node has not had its UC upgraded.
• The Upgrade Coordinator(UC) version is confirmed via API, in this example NSX is being upgraded from 4.2.0.0 to 4.2.2.1.

GET https://<problematic manager node>/api/v1/upgrade/summary  --> Problem manager node

{
    "system_version": "4.2.0.0.0.24105817",
    "upgrade_coordinator_version": "4.2.0.0.0.24105821",   <<< UC version has not been upgraded to the target version
    "upgrade_status": "PAUSED",
    "target_version": "4.2.2.1.0.24765084",
    "component_target_versions": [
        {
            "component_type": "EDGE",
            "target_version": "4.2.2.1.0.24765090"
        },
        {
            "component_type": "HOST",
            "target_version": "4.2.2.1.0.24765085"
        },
        {
            "component_type": "MP",
            "target_version": "4.2.2.1.0.24765087"
        }
    ],
    "upgrade_coordinator_updated": false    <<<< UC has not been upgraded
}

GET https://<non problematic Manager node>/api/v1/upgrade/summary  --> Good manager node which has the correct version of UC 

{
    "system_version": "4.2.0.0.0.24105817",
    "upgrade_coordinator_version": "4.2.2.1.0.24765084", <<<< UC matches the target version 
    "upgrade_status": "PAUSED",
    "target_version": "4.2.2.1.0.24765084",
    "component_target_versions": [
        {
            "component_type": "EDGE",
            "target_version": "4.2.2.1.0.24765090"
        },
        {
            "component_type": "HOST",
            "target_version": "4.2.2.1.0.24765085"
        },
        {
            "component_type": "MP",
            "target_version": "4.2.2.1.0.24765087"
        }
    ],
    "upgrade_coordinator_updated": true    <<<< UC has been upgraded
}

Environment

VMware NSX

Cause

The upgrade_coordinator_version did not get updated with the Target version and issue is seen as an error in the NSX Manager upgrade precheck

Resolution

This is a known issue impacting VMware NSX.

To resolve the issue:

1. Run the GET command against the problem node
   
   

   GET https://<problematic manager node>/api/v1/upgrade/summary

   {
    "system_version": "4.2.0.0.0.24105817",
    "upgrade_coordinator_version": "4.2.0.0.0.24105821",
    "upgrade_status": "PAUSED",
    "target_version": "4.2.2.1.0.24765084",   <<<<< Use this value in step 2
    "component_target_versions": [
        {
            "component_type": "EDGE",
            "target_version": "4.2.2.1.0.24765090"
        },
        {
            "component_type": "HOST",
            "target_version": "4.2.2.1.0.24765085"
        },
        {
            "component_type": "MP",
            "target_version": "4.2.2.1.0.24765087"  <<<< Do not use this value in step 2
        }
    ],
    "upgrade_coordinator_updated": false
}


2. Identify the UUID of the problem NSX Manager
   
   ssh as admin to any NSX Manager run the command get node, copy the UUID of the problem Manager
   
   
   
3. Use the below POST API to update the UC version on the problem node
   

    

   POST https://<problematic node_ip>/api/v1/cluster/<UUID of Manager from step 2>/node/upgrade/uc 

    

   Body:
   {
  "product_version": "4.2.2.1.0.24765084"  <<< value from step 1
}

    

4. Confirm the change has taken effect

 GET https://<problematic manager node>/api/v1/upgrade/summary

{
    "system_version": "4.2.0.0.0.24105817",
    "upgrade_coordinator_version": "4.2.2.1.0.24765084",    <<<< UC version now matching target version
    "pre_upgrade_bundle_version": "4.2.2.1.0.24765084",
    "upgrade_status": "PAUSED",
    "target_version": "4.2.2.1.0.24765084",
    "component_target_versions": [
        {
            "component_type": "EDGE",
            "target_version": "4.2.2.1.0.24765090"
        },
        {
            "component_type": "HOST",
            "target_version": "4.2.2.1.0.24765085"
        },
        {
            "component_type": "MP",
            "target_version": "4.2.2.1.0.24765087"
        }
    ],
    "upgrade_coordinator_updated": true
}

Additional Information

Related KBs:

ERROR: "UC is not updated on node" when performing prechecks on manager upgrade

"Check for the version sync status of UC on all the MP nodes" FAILURE seen for NSX precheck for upgrade

NSX upgrade pre-check fails with error "Pre-upgrade checks failed for MP: UC is not updated on node <NSX_MANAGER_Node(s)>. Please upgrade the UC on these nodes first and ensure the version of UC is same across all MP node"