NSX upgrade stuck while host upgrade is IN PROGRESS displaying "Not Started" on few hosts.

Products

• VMware NSX

Issue/Introduction

• In attempting to upgrade from NSX 3.2.X to 4.2.X.
• NSX-T upgrade to version is triggered from SDDC.
• NSX-T manager reports hosts component as IN Progress, few hosts as 100% success and few hosts status shows "Not started".
•  When the "esxcli software vib list" is run on the impacted ESXi hosts, the vibs reflect the target NSX version.
• The popup has been at the same percentage for a long time.
• In Manager logs - var/logs/upgrade-coordinator.log, shows the upgrade-status "Not started"

 025-##-12T17:01:43.724Z  INFO http-nio-#.#.#.#-7##2-e##c-4 AbstractHostUpgradeManager 29528 SYSTEM [ns###76 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [HUPS] Updated the  upgrade unit 5d####af-##e1-4b##-a##1-d4####e2cda2. Upgrade unit = UpgradeUnit [id=####f8af-####-4b13-ae01-d4####e2cda2, TransportNodeID =5d####8af-12##-4b13-a###-d492####cda2, name=, description=, type=HOST, upgradeUnitSubtype=RESOURCE, currentVersion=4.2.1.2.0.24476##0, warnings=[], errors=null, metaData={DEPLOYMENT_STATUS=INSTALL_SUCCESSFUL, HYPERVISOR_VERSION=7.0.3, PARENT_COMPLUTE_COLLECTION_DISPLAY_NAME=####-Cluster, IN_TN_MAINTENANCE_MODE=false, COMPUTE_MANAGER_VERSION=7.0, PARENT_COMPLUTE_COLLECTION=0a08####-eda8-46bf-aec9-####10d2f375:domain-##, VLCM_ENABLED=true, CM_LOCAL_ID=host-##57, DISCOVERED_IPS=X.X.X.X, COMPUTE_MANAGER_PORT=###, vlcm_sah=false, HYPERVISOR_OS=ESXI, HYPERVISOR_IP=X.X.X.X, COMPUTE_MANAGER_SERVER=#####.ffg, lockdownMode=lockdownDisabled, COMPUTE_MANAGER_ID=0a#####0-eda8-46bf-a###-8f74####2f375, COMPUTE_MANAGER_TYPE=vCenter}, rebooting=false, UaReportedStatus=NOT_SYNCED, extendedConfiguration=[KeyValuePair [key=UpgradeUnitPlatformType, value =Managed ESX, class=class com.#####.nsx.management.upgrade.model.KeyValuePair, hashCode=20#####6#]], progressTracker=UpgradeUnitProgressCollectorImpl [reference=5#####af-12e1-##13-a##1-d49####2cda2, referenceType=HOST, getProgressPercentage()=0, getUpgradeStatus()=NOT_STARTED, getLastProgressMessage()=null, getParent()=ReferencedProgressCollectorImpl [reference=0####00-##a8-46##-##c9-###410##f375:domain-c8, getProgressPercentage()=22, getUpgradeStatus()=PAUSED, getLastProgressMessage()=null]], class=class com.vmware.nsx.management.upgrade.model.UpgradeUnit, disabled=false, isSkipUpgrade=false, hashCode=-333####3]

Environment

VMware NSX-T Data Center
VMware NSX

Cause

The install-upgrade service is not responding. This may be due to a transient issue with the service updating the UI.

Resolution

• Restart the install-upgrade service on Orchestrator NSX Manager nodes. This causes no impact to datapath traffic.
• From the CLI admin prompt on node, run the following commands:
• Restart service install-upgrade
• Once completed, the upgrade status of each ESXi Hosts show as "Success"