NSX Upgrade Fails or Hangs Due to a Stale Transport Node Entry

Products

• VMware NSX

Issue/Introduction

An NSX upgrade initiated from the SDDC fails or hangs on a specific host. The upgrade process is unable to proceed to the next step because of an incomplete host upgrade.

Upon verification in the NSX UI, the host causing the failure is visible as an "unmanaged host." This typically occurs when a host has been previously decommissioned and removed from the SDDC and vCenter, but its registration was not fully cleared from NSX.

Environment

 

• VMware NSX

• VMware vCenter Server

• Software-Defined Data Center (SDDC)

 

Cause

The failure is caused by a stale transport node entry persisting within the NSX Manager database.

During an NSX upgrade, the upgrade process queries the NSX Manager database for all registered transport nodes and attempts to apply the upgrade to them. Because the decommissioned host is still registered in the NSX database (despite being physically or logically removed from the SDDC and vCenter), the upgrade fails when it is unable to communicate with or successfully update the stale node. The overarching upgrade sequence cannot proceed until this specific host's status is resolved.

Resolution

To resolve this issue, you must manually remove the stale transport node entry from the NSX Manager database using the NSX API and reset the host upgrade plan.

Follow these steps:

1. Force-Delete the Stale Transport Node
   
   Execute a force-delete of the stale transport node via the NSX Manager API. Run the following API call against the NSX Manager:
   
   DELETE https://<nsx-mgr-ip>/api/v1/transport-nodes/<transport-node-UUID>?force=true&unprepare_host=true
   
   Note: Ensure that this API call returns a 200 OK HTTP status code.
   
   
   
   
2. Verify the Node Removal
   
   Confirm that the node has been completely removed from the database by running an API GET command against the Transport Node ID (TNID):
   
   GET https://<nsx-mgr-ip>/api/v1/transport-nodes/<transport-node-UUID>
   
   Note: This command should return a 404 Not Found error, confirming the node no longer exists in the database.
   
   
   
   

3. Reset the Upgrade Plan

   Verify that the upgrade-coordinator service is currently running on the NSX Manager.

   Reset the host upgrade plan by running the following API call towards the NSX Manager VIP:
   
   POST https://<mgr_VIP>/api/v1/upgrade/plan?action=reset&component_type=HOST
   
   

4. After successfully resetting the upgrade plan, you can resume or reinitiate the NSX upgrade process.

Additional Information

Please check below KB for more details on resetting NSX upgrade plan:

Reset NSX Upgrade Prechecks