Deleted ESX host from vCenter blocks the NSX upgrade in VCF 9.1

Products

• VMware SDDC Manager / VCF Installer

Issue/Introduction

• An ESX host was removed from vCenter inventory (e.g., due to decommissioning or hardware failure).
• During a VCF 9.1 upgrade, the 'CONFIGURE' button in SDDC Manager is grayed out at the 'VMware NSX Edge Clusters 9.1.0.0 and NSX Upgrade Finalize' step, even though the NSX Manager nodes, vCenter, and ESX hosts have successfully upgraded to 9.1.
• The following error message is observed:
  "NSX Dataplane upgrade is not available for domain ... since all vCenter and ESX on the dependent domains are not upgraded."

         

• The following error messages are observed in lcm-debug.log:

     ..."<NSX_MANAGER>:null:null": { "failedCount": 0, "groupId": "########-####-####-####-############", "groupName": "Group 1 for ESXI", "percentComplete": 0.0, "status": "NOT_STARTED", "upgradeUnitCount": 1, "__dynamicStructureFields": { "fields": {}, "name": "" } }

     ...

     ..."errorPack":{"errorCode":"LCM_NSXT_HOST_NOT_UPGRADED_IN_SHARED_DOMAIN","arguments":["<DOMAIN_NAME>"],"referenceToken":"######"}

Environment

VCF 9.1

VMware NSX 9.1

Cause

A stale host transport node entry exists in the NSX Manager. 

Resolution

Delete the ESX host from NSX manager, retry the upgrade from VCF Operations. (See details in Manual removal of hosts from VC without removing NSX from them at first have resulted in stale host entries on the NSX appliance)