VCF Upgrade Blocked by Duplicate UNKNOWN NSX Transport Node After Out-of-Band Remediation

Products

• VMware NSX

Issue/Introduction

VMware Cloud Foundation (VCF) domain upgrade is blocked and cannot complete. During the migration from a standalone vSphere and NSX environment to VCF 9.0.2.0, ESXi hosts were remediated manually out-of-band via vCenter Server instead of VCF Operations.

Following the remediation, a duplicate ESXi host appears in the NSX inventory reporting an "UNKNOWN" status, while the healthy, upgraded host appears clean with SUCCESS status in the cluster.

This blocked state causes VCF upgrade pre-checks to fail, leaving the VCF domain suspended at the older version.

Environment

VMware Cloud Foundation 5.2.2 / 9.0.2.0
VMware NSX 4.3.2
VMware vCenter Server 8.0 U3
VMware ESXi 8.0 U3

Cause

Remediating the ESXi host directly via vCenter Server out-of-band alters the host's MoRef ID/UUID. NSX registers the updated host as a new object while retaining the previous state as an orphaned, unmanageable Transport Node in its database.

Resolution

Workaround:

1. Access the NSX Manager UI and navigate to System > Fabric > Nodes.

2. Identify the duplicate ESXi host reporting the "UNKNOWN" state and extract its Transport Node UUID.

3. Using an API client (e.g., cURL) authenticated as an NSX administrator, execute a forceful DELETE request against the orphaned UUID. The force=true and unprepare_host=false parameters are mandatory to ensure only the stale database entry is removed without affecting the active host.

   curl -k -u admin:'<password>' -X DELETE "https://<NSX-Manager-IP>/api/v1/transport-nodes/<Orphaned-Transport-Node-UUID>?force=true&unprepare_host=false"
   

4. Allow approximately 5 minutes for database synchronization across the NSX Manager cluster.

5. Verify the deletion by executing a GET request against the same UUID. It must return a 404 NOT_FOUND status.

   curl -k -u admin:'<password>' -X GET "https://<NSX-Manager-IP>/api/v1/transport-nodes/<Orphaned-Transport-Node-UUID>/state"
   

6. Return to SDDC Manager or the NSX Upgrade GUI, acknowledge the previously blocked state, and re-run the VCF upgrade pre-checks.

Additional Information

• NSX upgrade pre-check fails for host

• ESXi host status Unknown: NSX UI opsAgent crash