NSX-T upgrade in VCF fails with: Failed to collect edge cluster node host info maps

Products

• VMware NSX

Issue/Introduction

• You are upgrading NSX-T from within VCF.
• The upgrade fails with error like the following in the VCF log: lcm/thirdparty/upgrades/<update-attempt-uuid>/sddcmanager-migration-app/logs/sddcmanager_migration_app_upgrade.log

YYYY-MM-DD ERROR [,,,] 14851 --- [ool-2-thread-13] c.v.e.s.o.model.error.ErrorFactory : [8RIT02] EDGE_NODE_HOST_INFO_COLLECTION_FAILED Failed to collect edge cluster node host info maps
...
at java.lang.Thread.run(Thread.java:748) [na:1.8.0-internal] Caused by: java.lang.IllegalArgumentException: Host host-## not found

• Using the NSX-T REST API /api/v1/fabric/nodes in the results we can see the Edge node is referring to the host from the error above in the host-id section:

...
            "host_id": "host-##",
            "hostname": "########.####.local",
...

Note: In the above message host-XX, will refer to a specific vCenter host MOID, to see the MOID.
The UUID for each upgrade will change, to find the most recent, go to lcm/thirdparty/upgrades/ and run ls -ltr to find the most recent upgrade attempt.

Environment

VMware NSX-T Data Center 3.x
VMware NSX 4.x
VMware Cloud Foundation

Cause

This happens when the edge node still refers to the ESXi host, even after that host has been removed from vCenter.
This may occur due to an incorrect process used to remove an ESXi host in a VCF/NSX environment.

Resolution

Use the correct process to remove ESXi host from VCF and NSX environments.

NSX process: Best Practice for Decommissioning an NSX-Prepared ESXi Host to Avoid Stale Entries

VCF process: Decommission ESX Hosts

Workaround:
Replace the NSX Edge node that is referring to the no longer present ESXi host: Replace an NSX Edge Transport Node Using the NSX Manager UI