NSX Manager node stuck in "Upgrade Mode" during VCF 9.1.x upgrade
Article ID: 450811
Updated On:
Products
Issue/Introduction
During the patch upgrade of a VMware Cloud Foundation (VCF) environment to version 9.1.x, the NSX Manager cluster upgrade may stall. While the vCenter Server upgrade completes successfully, one or more NSX Manager nodes may become unresponsive or remain stuck in "Upgrade Mode" within the Life Cycle management view.
The NSX cluster upgrade does not proceed past the Management Plane stage.
A specific NSX Manager node remains offline and is labeled as "stuck in upgrade mode" in the UI.
The upgrade stalled at the
switch_osstep due to a mount path assertion error (AssertionError: Could not determine device because path is not mounted).Attempts to revalidate or progress the workflow via SDDC Manager or NSX UI fail to resolve the state.
Exact Error Message: Unable to upgrade cluster. Node <Node_Name> is stuck in upgrade mode and will not come online.
get upgrade progress-statusshows:
Environment
VMware Cloud Foundation 9.1.x
VMware NSX 9.1.x
Cause
This issue occurs due to a failure in the automated upgrade playbook execution on the individual manager node.
Resolution
To resolve this issue, the upgrade playbook must be manually replayed on the affected appliance to complete the stalled tasks.
- Log in to the affected NSX Manager node (the one stuck in upgrade mode) as root.
- Manually replay the upgrade playbook by identifying the specific bundle and running the following command:
"start upgrade-bundle <bundle-name> playbook VMware-NSX-manager-<version>-playbook" - Check the upgrade process by running "
get upgrade process-status" - Monitor the CLI output until the playbook execution completes successfully.
- Navigate to the NSX Manager UI:
- Go to System > Upgrade.
- Select Continue to allow the Upgrade Coordinator to verify the node status and proceed with the remaining cluster nodes.
Additional Information
Feedback
