Incomplete NSX-T upgrade - upgrade pre-check fails post Node OS upgrade with error "Pre-upgrade checks failed for MP: UC is not updated on node NSX_MANAGER. Please upgrade the UC on these nodes first and ensure the version of UC is same across all MP node

Products

• VMware NSX

Issue/Introduction

• VCF NSX-related operations fail with missing entry errors
• /var/log/upgrade-coordinator/upgrade-coordinator.log
  
  2023-07-04T14:12:04.566Z ERROR task-executor-0-workitem-MP-VpnSectionCleanupMigrationTask WorkItem 69500 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30062" level="ERROR" subcomp="upgrade-coordinator"] Error encountered while upgrading upgrade unit VpnSectionCleanupMigrationTask
com.vmware.nsx.management.upgrade.plugin.mp.exceptions.MPUpgradeException: null
        at com.vmware.nsx.management.upgrade.plugin.mp.service.MPRollingUpgradeServiceImpl.trackMigration(MPRollingUpgradeServiceImpl.java:913) ~[libnsx-upgrade-plugins.jar:?]
        at com.vmware.nsx.management.upgrade.plugin.mp.service.MPRollingUpgradeServiceImpl.upgradeMPMigration(MPRollingUpgradeServiceImpl.java:776) ~[libnsx-upgrade-plugins.jar:?]
        at com.vmware.nsx.management.upgrade.plugin.mp.service.MPRollingUpgradeServiceImpl.upgrade(MPRollingUpgradeServiceImpl.java:230) ~[libnsx-upgrade-plugins.jar:?]
        at com.vmware.nsx.management.upgrade.plugin.mp.MpUpgradePlugin.upgrade(MpUpgradePlugin.java:265) ~[libnsx-upgrade-plugins.jar:?]

 

2023-07-04T23:47:19.496Z  INFO http-nio-127.0.0.1-7442-exec-3 UpgradeServiceImpl 69500 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Triggering POST Upgrade for component MP ...

2023-07-05T00:40:43.348Z  INFO localhost-startStop-1 UpgradeServiceImpl 3279 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Triggering POST Upgrade for component MP ...

2023-07-07T05:03:02.945Z  INFO localhost-startStop-1 UpgradeServiceImpl 2952 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Triggering POST Upgrade for component MP ...

• For further investigation /var/log/upgrade-coordinator/logical-migration.log will not be updated , this is due to log file permission issue which is addressed in NSX 4.1.1

Environment

VMware NSX 4.0.0.1
VMware NSX-T Data Center 3.x

Cause

MP upgrade failed at VpnSectionCleanupMigrationTask. However, it got marked as successful incorrectly due to reset MP plan invoked after failure. Reset plan API called for MP component, caused plan type to set as non-rolling upgrade, and upgrade to get marked as successful incorrectly

If Upgrade coordinator is not restarted after reset plan exception occurs due to MP upgrade is not PAUSED

On VCF setups, reset plan is called after MP upgrade, and VCF status is already marked as successful

Resolution

This issue is resolved in VMware NSX 4.1.1, available at Broadcom downloads.

Workaround:

Restart Upgrade Coordinator service on all 3 MP nodes

 

Additional Information

If you believe you have encountered this issue and the workaround does not resolve the issue, please open a support case with Broadcom Support and refer to this KB article.

For more information, see Creating and managing Broadcom support cases.