VMware NSX-T Manager upgrade failed and is stuck "in-progress" for an indefinite time

Products

• VMware NSX

Issue/Introduction

 

• NSX Manager upgrade fails and is stuck in-progress for infinite time.
• In the NSX-T Manager log (/var/log/upgrade-coordinator/upgrade-coordinator.log) you will entries similar to:

2020-12-05T13:43:43.503Z INFO http-nio-127.0.0.1-7442-exec-2 UpgradeServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Triggering upgrade of MP component
2020-12-05T13:50:39.942Z INFO task-executor-0-workitem-MP-99740942-9437-####-####-########b61 MPUpgradeServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [MP UCP] Issuing step 6 Reboot MP node: MPClusterNodeInfo{nodeId=99740942-9437-####-####-########b61, ip=192.168.10.38, status=UP, mpaClientId=cvn-mp-mpa-af5e8332-47d1-####-####-########4b7, self=false, nsxVersion=2.5.2.0.0.16615906}
2020-12-05T13:50:41.978Z INFO task-executor-0-workitem-MP-99740942-9437-####-####-########b61 UpgradeAgentMessagingServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Issuing reboot succeeded on 99740942-9437-####-####-########b61. Info from Upgrade Agent:
2020-12-05T13:50:41.978Z INFO task-executor-0-workitem-MP-99740942-9437-####-####-########b61 UpgradeAgentMessagingServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Polling for reboot ae7ab81c-a470-####-####-########f9e on 99740942-9437-####-####-########b61 with timeout 1797964
2020-12-05T14:20:39.944Z ERROR task-executor-0-workitem-MP-99740942-9437-####-####-########b61 UpgradeAgentMessagingServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30033" level="ERROR" subcomp="upgrade-coordinator"] Polling reboot timedout on 99740942-9437-####-####-########b61
2020-12-05T14:20:39.944Z ERROR task-executor-0-workitem-MP-99740942-9437-####-####-########b61 MPUpgradeServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30426" level="ERROR" subcomp="upgrade-coordinator"] Error in MP Upgrade while instructing UA on MP node MPClusterNodeInfo{nodeId=99740942-9437-####-####-########b61, ip=192.168.10.28, status=UP, mpaClientId=cvn-mp-mpa-af5e8332-47d1-####-####-########4b7, self=false, nsxVersion=2.5.2.0.0.16615906} to reboot system
com.vmware.nsx.management.upgrade.exceptions.UpgradeAgentMessagingServiceException: Polling reboot timed out
        at com.vmware.nsx.management.upgrade.common.UpgradeAgentMessagingServiceImpl.checkUAResponseSuccess(UpgradeAgentMessagingServiceImpl.java:398) ~[uc-core-1.0.jar:?]

Environment

VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center

Resolution

This issue has been fixed in NSX-T 2.5.3.  available at  Broadcom Downloads.

Workaround:

Restart the upgrade-coordinator service and retry the upgrade.

 

Additional Information

Impact/Risks:

The upgrade will be stuck in-progress indefinitely.