NSX Host upgrade marked as PAUSED after successful NSX upgrade

Products

• VMware NSX

Issue/Introduction

• NSX upgrade marked as PAUSED in the UI.
• User won't able to start a new upgrade moving forward until current status is reconciled to success.
• Following log entries seen in /var/log/upgrade-coordinator/upgrade-coordinator.log:

INFO http-nio-127.0.0.1-7442-exec-1 HostFinderServiceImpl 8364 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Since transport node status is not available for ########-####-####-####-############. Creating status as UNKNOWN.

INFO http-nio-127.0.0.1-7442-exec-1 UpgradeServiceImpl 11081 SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Component type = HOST , Upgrade Status = PAUSED, Progress Percentage = 98 , Progress Message = null

Environment

VMware NSX

Cause

After an upgrade completes, there are some Transport Nodes disconnected from NSX. As Upgrade Coordinator is routinely checking for the status of the Transport Nodes in the system, it is unable to learn their latest status and hence marks those disconnected Transport Nodes as UNKNOWN. As a result, the overall upgrade status to retrogress to PAUSED. 

Resolution

This issue is resolved in NSX version 4.1.2 and above.

Workaround:

• Make sure that the TNs for which the upgrade was marked as paused are back to healthy state, or they have been deleted and no stale record still exists in NSX inventory.
• Then re-trigger the following below API:

GET /api/v1/upgrade/upgrade-units-stats?sync=true

This API will make Upgrade Coordinator re-sync upgrade status based on the latest system state