NSX UI shows Upgrade as Paused despite successful completion
search cancel

NSX UI shows Upgrade as Paused despite successful completion

book

Article ID: 394172

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • The upgrade may have fully completed and displayed as successful in the past. When preparing to upgrade again or otherwise checking the Upgrade page in NSX Manager's UI, it shows the previous upgrade in failed state.
  • This can also occur during the Host upgrade portion of an ongoing upgrade.
  • The overall upgrade status appears paused at 99% in the NSX UI and clicking on Continue does not resolve the issue.
  • Some transport nodes (TNs) may show the upgrade as stuck at 60%.
  • Transport nodes that have been decommissioned post-upgrade may display an upgrade status of paused at 0%.
  • Log entries in /var/log/upgrade-coordinator/upgrade-coordinator.log show messages about HOST components with PAUSED status similar to the following: 
    2025-03-10T08:49:39.423Z  INFO http-nio-127.0.0.1-7442-exec-5 PluginManager 3204 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Invoking sync plan for component HOST
2025-03-10T08:52:05.834Z  INFO http-nio-127.0.0.1-7442-exec-5 ReferencedProgressCollectorImpl 3204 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] For ########-####-####-####-############status is NOT_STARTED, Total 14, Success 0, Fail 0, Paused 0, Not-started 14
2025-03-10T08:52:05.851Z  INFO http-nio-127.0.0.1-7442-exec-5 UpgradeServiceImpl 3204 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Component type = HOST , Group = ########-####-####-####-############, Upgrade Status = NOT_STARTED, Progress Percentage = 0
2025-03-10T08:52:06.636Z  INFO http-nio-127.0.0.1-7442-exec-5 ReferencedProgressCollectorImpl 3204 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] For HOST status is PAUSED, Total 52, Success 51, Fail 0, Paused 0, Not-started 1
2025-03-10T08:52:06.636Z  INFO http-nio-127.0.0.1-7442-exec-5 UpgradeServiceImpl 3204 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Component type = HOST , Upgrade Status = PAUSED, Progress Percentage = 97 , Progress Message = null
  • For situations where a previous upgrade had completed successfully before running into this issue, the upgrade-coordinator.log would also show messages like: 
    2025-02-21T21:51:50.247Z ERROR http-nio-127.0.0.1-7442-exec-4 HostUpgradePlugin 3204 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30145" level="ERROR" subcomp="upgrade-coordinator"] [HUT] For Upgrade unit ########-####-####-####-############ , deployment status is POWERED_OFF , expected was INSTALL_SUCCESSFUL
2025-02-21T21:51:50.247Z ERROR http-nio-127.0.0.1-7442-exec-4 HostUpgradePlugin 3204 SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30145" level="ERROR" subcomp="upgrade-coordinator"] [HUT] For Upgrade unit ########-####-####-####-############ , deployment status is POWERED_OFF , expected was INSTALL_SUCCESSFUL


Environment

VMware NSX

Cause

The upgrade status in the NSX UI appears paused or incomplete due to the presence of stale host Transport Node data in NSX Manager. These hosts were not fully removed from the NSX inventory, causing the host upgrade status to regress or to stall, in the case of an ongoing upgrade where any TN is orphaned after the upgrade begins. As a result, the system continues to reflect an incomplete upgrade status that cannot be resolved with the Continue button even if the upgrade options for hosts are modified to attempt to skip problem ones.

Resolution

Removing or force deleting NSX from decommissioned hosts that appear in the NSX Manager UI allows the system to reconcile the upgrade status correctly and mark it as successful.

Refer to instructions in Attempting to Remove NSX for an ESXi host Transport Node in NSX Manager fails when the host is not available in vSphere.

Additional Information

Note that a similar condition may occur where ESXi host Transport Nodes are still present in vSphere but are in a powered off state. If that is the case, powering the host back on and selecting to Continue the upgrade should move it to a Successful state.

Powered by Wolken Software