NSX Upgrade State Shows Previous Upgrade Still in Progress, During Host Upgrade Phase.

Products

• VMware NSX

Issue/Introduction

While initiating the NSX upgrade (System->Upgrade) process, the upgrade module indicates that a previous upgrade is still in progress, and the upgrade summary shows that the 'Upgrade Hosts' phase is incomplete. The current version and target version showing the same. As shown below:

 

When you click 'Continue' to check the full status, you will see that the Edge upgrade has completed successfully. However, the 'Hosts upgrade' phase is paused for one of the host clusters, even though all the members in that cluster are showing as fully upgraded. Meanwhile, the 'Manager upgrade' phase has also completed, which is the next phase following the hosts upgrade in the NSX upgrade process. As shown below:

Environment

VMware NSX 

Cause

The cluster that participated in the previous upgrade, including all the hosts that were part of it, has been removed. As a result, neither the cluster nor its hosts are part of NSX anymore. However, this has left behind a stale entry, which is causing the issue.

Resolution

 

• Check hosts status using below GET API, that may show up the host disconnected/down:

GET https://<MPIP>/api/v1/transport-nodes/<Transport-Node-UUID>/state

• Check and confirm in vCenter that this cluster and its hosts are no longer present.
• Once confirmed that the hosts are no longer in use, in vCenter, execute the DELETE API below to remove these stale entries for the hosts:

DELETE https://<MPIP>/api/v1/transport-nodes/<Transport-Node-UUID>?force=true&unprepare_host=false

• Wait for 1-5 minutes to allow time for the cleanup to occur, then use the GET API to confirm that the host are no longer present in NSX:

GET https://{{MPIP}}/api/v1/transport-nodes/<Transport-Node-UUID>/state

• Check and verify the upgrade status, it should now indicate that the hosts upgrade phase is clear, with no hosts or clusters in a paused state.
• If necessary, reboot the manager. The previous upgrade status should now be in a completed state, allowing us to proceed with the new upgrade.