NSX Upgrade State Shows Previous Upgrade Still in Progress, During Host Upgrade Phase.

Products

• VMware NSX

Issue/Introduction

• While initiating the NSX upgrade (System->Upgrade) process, the upgrade module indicates that a previous upgrade is still in progress, and the upgrade summary shows that the 'Upgrade Hosts' phase is incomplete.
• The current version and target version showing the same as shown below:

 

• The 'Hosts upgrade' phase is paused for one of the host clusters, even though all the members in that cluster are showing as fully upgraded.

Environment

VMware NSX 

Cause

• The cluster that participated in the previous upgrade, including all the hosts that were part of it, has been removed.
• As a result, neither the cluster nor its hosts are part of NSX anymore. However, this has left behind a stale entry, which is causing the issue.

Resolution

The following API calls needs to be executed to remove the stale entry of the Transport Node that is no longer in use. API calls can be executed either via 3rd party tools such as postman or by logging in as root into the NSX manager via putty/ssh

• Check hosts status using below GET API, that may show up the host as disconnected or down:

GET https://<NSX-Manager-FQDN>/api/v1/transport-nodes/<Transport-Node-UUID>/state

OR

curl -X  GET -H 'Content-Type: application/json' -sN -k -u admin  "https://<NSX-Manager-FQDN>/api/v1/transport-nodes/<Transport-Node-UUID>/state"

• Login to the vCenter UI and verify that the host is no longer present by searching the vCenter inventory using the transport node hostname or IP.
• Try pinging the host FQDN or IP from the NSX manager or vCenter to further ensure that the host is not in use.
• Once confirmed that the hosts is no longer in use, run the DELETE API below to remove the stale entry of the host.

DELETE https://<NSX-Manager-FQDN>/api/v1/transport-nodes/<Transport-Node-UUID>?force=true&unprepare_host=false

OR

curl -X  GET -H 'Content-Type: application/json' -sN -k -u admin  "https://<NSX-Manager-FQDN>/api/v1/transport-nodes/<Transport-Node-UUID>?force=true&unprepare_host=false"

• Login into all the NSX managers in the cluster as user admin and run the command start search resync all to ensure the indexes are in sync on all NSX managers inventory.
• Log out of NSX-T UI and log back in (reindexing may take a few minutes)
• Check NSX-T search for host name in question to ensure no UUID shows up.
• The status of the deleted transport node can be further confirmed via GET API to confirm that the host are no longer present in NSX:

GET https://<NSX-Manager-FQDN>/api/v1/transport-nodes/<Transport-Node-UUID>/state

OR

curl -X  GET -H 'Content-Type: application/json' -sN -k -u admin  "https://<NSX-Manager-FQDN>/api/v1/transport-nodes/<Transport-Node-UUID>/state"

• Check and verify the upgrade status on the NSX Manager UI System --> Upgrade, which should now allow to proceed with the upgrade.
  
  

If you still face the issue, kindly open a case with Broadcom Support Creating and managing Broadcom support cases

Additional Information

To be able to access the NSX Manager as root , login to the NSX manager as user admin via putty/ssh and switch to the root login by running the command st en