After restoration from MP upgrade backup, pre-check throws Error: UC is not updated on node <NODE_ID>. Please upgrade the UC on these nodes first and ensure the version of UC is same across all MP nodes

Products

• VMware NSX

Issue/Introduction

• Restored from backup taken before MP upgrade.
• Performed the following steps to restore the system.
  1. Deploy your Management Plane node with the same IP address that the backup was created from.
  2. Upload the upgrade bundle that you used at the beginning of the upgrade process.
  3. Upgrade the Upgrade Coordinator.
  4. Restore the backup taken during the upgrade process.
• Pre-check throws Error: UC is not updated on node <NODE_ID>. Please upgrade the UC on these nodes first and ensure the version of UC is same across all MP nodes
• UC log at /var/log/upgrade-coordinator/upgrade-coordinator.log throws:
  <TIMESTAMP>  WARN pool-##-thread-# UpgradeServiceImpl <INT> SYSTEM [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="upgrade-coordinator"] [PUC] Pre-upgrade check InspectionTaskInfo[id=ucVersionCheck,name=Check for the version sync status of UC on all the MP nodes.,description=This check ensures the version of UC is in sync across all the MP nodes before starting MP upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>] failed with result BasicInspectionTaskResult{status=FAILURE, taskInfo=InspectionTaskInfo[id=ucVersionCheck,name=Check for the version sync status of UC on all the MP nodes.,description=This check ensures the version of UC is in sync across all the MP nodes before starting MP upgrade.,componentType=MP,needsAcknowledgement=false,acknowledgement=false,needsResolution=false,resolution=false,resolutionError=<null>], failureMessages=null, failures=[{"moduleName":"upgrade-coordinator","errorCode":30486,"errorMessage":"UC is not updated on node <Nodes>. Please upload the upgrade bundle and upgrade UC on one of these nodes to ensure the version of UC is same across all MP nodes."}]}

Environment

NSX-T Data Center 3.2
NSX 4.x

Cause

The version of UC must be same across all NSX Manager nodes.
Upgrade Coordinator (UC) is upgraded when MUB bundle is uploaded.
When new NSX Manager nodes are added by restore after UC upgrade, UC version on new nodes are not upgraded.

Resolution

Upgrade UC by uploading the same MUB bundle before proceed upgrade after new NSX Manager nodes are added.

Correct steps are:

1. Deploy your Management Plane node with the same IP address that the backup was created from.
2. Upload the upgrade bundle that you used at the beginning of the upgrade process.
3. Upgrade the Upgrade Coordinator.
4. Restore the backup taken during the upgrade process.
5. Upgrade the Upgrade Coordinator with the same MUB bundle used in Step 3 on newly added nodes.

Upgrade the Upgrade Coordinator