NSX-T Manager upgrade failed and stuck "in-progress" for an indefinite time

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• NSX Manager upgrade will fail and it will stuck in-progress for infinite time
• In the NSX-T Manager log (/var/log/upgrade-coordinator/upgrade-coordinator.log) you will entries similar to:

2020-12-05T13:43:43.503Z INFO http-nio-127.0.0.1-7442-exec-2 UpgradeServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Triggering upgrade of MP component
2020-12-05T13:50:39.942Z INFO task-executor-0-workitem-MP-99740942-9437-d2fd-17ee-862d71789b61 MPUpgradeServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] [MP UCP] Issuing step 6 Reboot MP node: MPClusterNodeInfo{nodeId=99740942-9437-d2fd-17ee-862d71789b61, ip=192.168.10.38, status=UP, mpaClientId=cvn-mp-mpa-af5e8332-47d1-455a-83e6-88ae58ab34b7, self=false, nsxVersion=2.5.2.0.0.16615906}
2020-12-05T13:50:41.978Z INFO task-executor-0-workitem-MP-99740942-9437-d2fd-17ee-862d71789b61 UpgradeAgentMessagingServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Issuing reboot succeeded on 99740942-9437-d2fd-17ee-862d71789b61. Info from Upgrade Agent:
2020-12-05T13:50:41.978Z INFO task-executor-0-workitem-MP-99740942-9437-d2fd-17ee-862d71789b61 UpgradeAgentMessagingServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" level="INFO" subcomp="upgrade-coordinator"] Polling for reboot ae7ab81c-a470-4ec6-ae55-9134e2678f9e on 99740942-9437-d2fd-17ee-862d71789b61 with timeout 1797964
2020-12-05T14:20:39.944Z ERROR task-executor-0-workitem-MP-99740942-9437-d2fd-17ee-862d71789b61 UpgradeAgentMessagingServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30033" level="ERROR" subcomp="upgrade-coordinator"] Polling reboot timedout on 99740942-9437-d2fd-17ee-862d71789b61
2020-12-05T14:20:39.944Z ERROR task-executor-0-workitem-MP-99740942-9437-d2fd-17ee-862d71789b61 MPUpgradeServiceImpl - SYSTEM [nsx@6876 comp="nsx-manager" errorCode="MP30426" level="ERROR" subcomp="upgrade-coordinator"] Error in MP Upgrade while instructing UA on MP node MPClusterNodeInfo{nodeId=99740942-9437-d2fd-17ee-862d71789b61, ip=192.168.10.28, status=UP, mpaClientId=cvn-mp-mpa-af5e8332-47d1-455a-83e6-88ae58ab34b7, self=false, nsxVersion=2.5.2.0.0.16615906} to reboot system
com.vmware.nsx.management.upgrade.exceptions.UpgradeAgentMessagingServiceException: Polling reboot timed out
        at com.vmware.nsx.management.upgrade.common.UpgradeAgentMessagingServiceImpl.checkUAResponseSuccess(UpgradeAgentMessagingServiceImpl.java:398) ~[uc-core-1.0.jar:?]

Environment

VMware NSX-T Data Center 2.5.x
VMware NSX-T Data Center

Resolution

• This issue has been fixed in NSX-T 2.5.3

Workaround:

• Restart the upgrade-coordinator service and retry the upgrade

Additional Information

Impact/Risks:

• The upgrade will be stuck "in-progress" indefinitely