VCF 9.1 NSX Edge Dataplane Upgrade and Finalize Cannot be Started (Stale Host Entry)
Article ID: 444026
Updated On:
Products
Issue/Introduction
- During a VMware Cloud Foundation (VCF) 9.1 upgrade, the Configure button in SDDC Manager is grayed out at the VMware NSX Edge Clusters 9.1.0.0 and NSX Upgrade Finalize step. This occurs even if NSX Manager nodes, vCenter, and ESXi hosts have successfully upgraded to version 9.1.
NSX Dataplane upgrade is not available for domain [domain-name] since all vCenter and ESX on the dependent domains are not upgraded. To complete the update of an NSX instance you must add the vCenter and ESX components to the update plan of this and all dependent domains. After you update the vCenter, ESX, and NSX Manager components, you can finalize the update.
Querying the NSX API for the upgrade summary (GET /api/v1/upgrade/summary) shows the upgrade_status is PAUSED (target 9.1.0.0100.25470810).
Querying the NSX API for host upgrade units (GET /api/v1/upgrade/upgrade-units?component_type=HOST) returns a higher number of units than actual physical hosts (e.g., returning 17 units for 16 real host transport nodes), indicating a duplicate host entry (e.g., esx06.abc.cde appears twice).
- The lcm-debug.log on SDDC Manager shows the following error code: errorCode:
..."<NSX_MANAGER>:null:null": { "failedCount": 0, "groupId": "########-####-####-####-############", "groupName": "Group 1 for ESXI", "percentComplete": 0.0, "status": "NOT_STARTED", "upgradeUnitCount": 1, "__dynamicStructureFields": { "fields": {}, "name": "" } }
...
..."errorPack":{"errorCode":"LCM_NSXT_HOST_NOT_UPGRADED_IN_SHARED_DOMAIN","arguments":["<DOMAIN_NAME>"],"referenceToken":"######"}
- While Broadcom KB 446110 addresses this identical error caused by a complete lack of configured Transport Nodes, this specific scenario occurs because a stale node record persists even though all active hosts were successfully upgraded to 9.1.
Environment
- VMware Cloud Foundation 9.1
- VMware NSX 9.1
Cause
A stale or orphaned NSX Transport Node record persists in the NSX database. This typically happens when an ESXi host was decommissioned and recommissioned (or renamed) without a complete cleanup in NSX. The VCF Lifecycle Management (LCM) service detects this orphaned record as an unupgraded unit, which blocks the finalization of the NSX upgrade.
Resolution
- Log in to the NSX Manager user interface.
- Navigate to System > Fabric > Nodes > Host Transport Nodes.
- Locate the Other Nodes section.
- Identify the stale host entry (often showing as Disconnected or Orphaned with a version lower than 9.1).
- Select the stale host and click Remove NSX (or Force Delete if prompted).
- Verify the record is removed by running the following API call on NSX Manager: GET /api/v1/upgrade/upgrade-units?component_type=HOST. Confirm the count of upgrade units matches the physical host count.
- Return to the SDDC Manager upgrade wizard. The Configure button is now available to proceed with the NSX Edge dataplane upgrade and finalization.
If the issue persists, contact Broadcom Support. See for assistance.
See details in Manual removal of hosts from VC without removing NSX from them at first have resulted in stale host entries on the NSX appliance
Additional Information
Feedback
