NSX upgrade cancels automatically in VCF Operations 9.0.2
Article ID: 431174
Updated On:
Products
Issue/Introduction
When you attempt to upgrade NSX from version 9.0.1.0 to 9.0.2.0 using VCF Operations (Fleet Management), the upgrade cancels automatically shortly after being scheduled.
All prechecks pass successfully, and no explicit error messages are presented in the user interface to explain the cancellation. Additionally, clearing SDDC manager database locks does not resolve the issue.
Environment
- VCF Operations 9.0.x
- VMware NSX 9.0.1.0
- Additional VMware NSX 4.2.3 instances registered to the same Management vCenter
Cause
This issue occurs due to a conflict between the Fleet Management UI selection and the underlying SDDC Manager inventory when multiple NSX instances are registered to the same Management vCenter.
In VCF Operations 9.0.x, the upgrade engine performs a global scan of the registered Compute Managers in vCenter. Even if you select only the NSX 9.0.1.0 instance for upgrade, the orchestration engine detects the additional NSX 4.2.3 instances and cancels the upgrade task to prevent an inconsistent state. The cancellation is driven by the following factors:
- Unsupported Topology (Multi-NSX): VCF 9.0.x deprecates the Multi-NSX topology. If the 4.2.3 managers are still registered as Compute Managers, the upgrade script detects an unsupported topology and silently cancels the task before an error is written to the UI.
- Version Precedence: VCF 9.0.x is designed to bring all managed components to a unified baseline. If the SDDC Manager has downloaded metadata for NSX 9.0.2, the orchestrator refuses to proceed if it detects other NSX Managers (4.2.3) that it believes should be part of a higher target baseline.
- Inventory Locking: SDDC Manager locks the entire NSX Cluster or Network resource during an upgrade. It attempts to validate the upgrade path for all detected NSX Managers. Because the upgrade path from 4.2.3 to 9.x can have specific blocking issues, the validation fails and the workflow cancels.
Log analysis from the NSX Manager (upgrade-coordinator logs) confirms that while the upgrade request is received from VCF, the execution status for the NSX components never initiates locally and remains stuck in a "NOT_STARTED" state.
INFO ... Got upgrade status summary overallUpgradeStatus: NOT_STARTED INFO ... Component: MP, status: NOT_STARTED, % complete: 0.0, details: null, canSkip: false INFO ... Component: EDGE, status: NOT_STARTED, % complete: 0.0, details: null, canSkip: false INFO ... Component: HOST, status: NOT_STARTED, % complete: 0.0, details: null, canSkip: false INFO ... Component: FINALIZE_UPGRADE, status: NOT_STARTED, % complete: 0.0, details: null, canSkip: false
Resolution
To resolve this issue, you can perform the NSX upgrade directly through the NSX Manager interface, bypassing VCF Operations (Fleet Management).
Note: This procedure is a workaround. Performing an out-of-band upgrade is not the expected behavior. If this issue occurs again during future upgrades or you would prefer not to use this workaround, please contact Broadcom Support so that Product Engineering can determine the root cause of the scheduling failure.
For detailed steps on performing the manual NSX upgrade, see the Manual Upgrade to NSX 9 by Using Upgrade Coordinator documentation.
Once the out-of-band upgrade is complete:
- Log in to VCF Operations.
- Perform a full inventory sync.
- Allow the SDDC health check tasks from Fleet Management > Tasks to run successfully.
- Verify that all components now reflect version 9.0.2.0 in the SDDC dashboard.
Additional Information
Feedback
