NSX upgrade pre-check fails with error "Pre-upgrade checks failed for MP: UC is not updated on node <NSX_MANAGER_Node(s)>. Please upgrade the UC on these nodes first and ensure the version of UC is same across all MP node"

Products

• VMware NSX

Issue/Introduction

• VCF NSX-related operations fail with missing entry errors
• This error occurs post Node OS upgrade 
• From the NSX manager logs, /var/log/upgrade-coordinator/upgrade-coordinator.log contains the following entries:
  
  2023-07-04T14:12:04.566Z ERROR task-executor-0-workitem-MP-VpnSectionCleanupMigrationTask WorkItem 69500 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30062" level="ERROR" subcomp="upgrade-coordinator"] Error encountered while upgrading upgrade unit VpnSectionCleanupMigrationTask
com.vmware.nsx.management.upgrade.plugin.mp.exceptions.MPUpgradeException: null
        at com.vmware.nsx.management.upgrade.plugin.mp.service.MPRollingUpgradeServiceImpl.trackMigration(MPRollingUpgradeServiceImpl.java:913) ~[libnsx-upgrade-plugins.jar:?]
        at com.vmware.nsx.management.upgrade.plugin.mp.service.MPRollingUpgradeServiceImpl.upgradeMPMigration(MPRollingUpgradeServiceImpl.java:776) ~[libnsx-upgrade-plugins.jar:?]
        at com.vmware.nsx.management.upgrade.plugin.mp.service.MPRollingUpgradeServiceImpl.upgrade(MPRollingUpgradeServiceImpl.java:230) ~[libnsx-upgrade-plugins.jar:?]
        at com.vmware.nsx.management.upgrade.plugin.mp.MpUpgradePlugin.upgrade(MpUpgradePlugin.java:265) ~[libnsx-upgrade-plugins.jar:?]

 

2023-07-04T23:47:19.496Z  INFO http-nio-127.0.0.1-7442-exec-3 UpgradeServiceImpl 69500 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Triggering POST Upgrade for component MP ...

2023-07-05T00:40:43.348Z  INFO localhost-startStop-1 UpgradeServiceImpl 3279 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Triggering POST Upgrade for component MP ...

2023-07-07T05:03:02.945Z  INFO localhost-startStop-1 UpgradeServiceImpl 2952 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Triggering POST Upgrade for component MP ...

 

Environment

VMware NSX 4.1.0 and earlier
VMware NSX-T Data Center 3.2.3 and earlier
VCF 5.x

Cause

This issue occurs when the management plane upgrade fails at the VPN Cleanup Migration Task (VPNSectionCleanupMigrationTask) and VCF triggers a reset of the management plane plan. This reset causes the plan type to be set as non-rolling upgrade in the NSX Manager, and the upgrade is incorrectly marked as successful. When the upgrade is continues, it fails with an error.

 

Note that the same error message can occur in situations where the symptoms don't match as identified above but the NSX Manager nodes really do not all have a matching Upgrade Coordinator version. This may happen after re-deploying one or more Manager nodes in a cluster after an incomplete upgrade, for example or due to any communication issue between the Manager nodes at the time of UC upgrade. 

Resolution

This issue is resolved in VMware NSX 4.1.1, available at Broadcom downloads.

Workaround:

Restart the install-upgrade service on the cluster orchestrator.

• Determine which manager has the orchestrator role by logging in to any of the NSX Managers as admin and issue the command:
  • get service install-upgrade (below is an example and will likely be different from your actual output)

nsxmgr01> get service install-upgrade
Thu Apr 10 2025 UTC 17:46:29.224
Service name:      install-upgrade
Service state:     running
Enabled on:        192.168.0.6

• Log in to the cluster orchestrator, identified in the previous command, as admin and restart the install-upgrade service
• • restart service install-upgrade

 

If the symptoms described in this article do not align but the alert about UC not being updated on nodes is present, attempt to resolve it by following the instructions provided in the alert:

• In the Updates page of the NSX Manager UI, navigate to Upgrade > Prepare in the central panel. Run Begin Upgrade from there. 
• The Begin Upgrade Progress steps should show various "Synchronizing..." components for the manager nodes. The Upgrade Coordinator service will also be stopped and started again during this process.
• Once that stage is complete, the upgrade should be able to proceed normally.

Additional Information

If you believe you have encountered this issue and the workaround does not resolve the issue, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.