NSX-T Management Plane upgrade failure, one node stuck in 90% and times out with upgrade failed

Products

• VMware NSX

Issue/Introduction

Symptoms:

• All services are down on the NSX Manager node with 90% upgrade progress.
• Attempting to retry the upgrade fails.
• The NSX-T Manager UI is unavailable.
• In /var/log/syslog.log similar entries to the below will be observed:

2022-01-16T20:28:03.932Z ngnynm-rochm02 NSX 1110 - [nsx@6876 comp="nsx-manager" subcomp="mpa-client" tid="1421" level="INFO"] [UPGRADE] MpaClientHostService Received: Msg type (com.vmware.nsx.upgrade_agent.PrepareUpgradeMsg)corelationId (e46e9fe9-xxxx-xxxx-xxxx-xxxxxxxxxxxx) from APH (d1dfaab2-xxxx-xxxx-xxxx-xxxxxxxxxxxx)

2022-01-16T20:28:03.933Z ngnynm-rochm02 NSX 1110 - [nsx@6876 comp="nsx-manager" subcomp="upgrade-agent" tid="1421" level="INFO"] Got PrepareUpgradeMsg()

2022-01-16T20:28:51.707Z ngnynm-rochm02 NSX 1110 - [nsx@6876 comp="nsx-manager" subcomp="upgrade-agent" tid="1421" level="INFO"] [SendMsg] Sending message (type:com.vmware.nsx.upgrade_agent.PrepareUpgradeResponseMsg, len:523)

2022-01-16T20:28:51.707Z ngnynm-rochm02 NSX 1110 - [nsx@6876 comp="nsx-manager" subcomp="upgrade-agent" tid="1421" level="ERROR" errorCode="MPA50004"] [SendMsg] failed to send message

Environment

VMware NSX-T Data Center
VMware NSX-T

Cause

This issue can occur if IP addressing and/or certificates in /etc/vmware/nsx/appliance-info.xml are different.
eg:

<appliance-info>
        <appliance-proxies>
                <appliance-proxy>
                        <uuid>13cXX8X1-XXXX-XXXX-XXX-XXXXXXXXXXX</uuid>
                        <ip>XX.XX.XX.121</ip>
                        <fqdn/>
                        <port>1234</port>

Resolution

Currently there is no resolution to this issue.

Workaround:
Should you encounter this issue, please raise a Support Request with VMware referencing this article.