NSX-T manager upgrade fails during skip-level upgrade from VCF 4.4 to VCF 4.5

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

NSX-T manager upgrade fails and the NSX-T manager cluster goes to degraded state.

Environment

Vmware Cloud Foundation 4.5
VMware Cloud Foundation 4.4

Resolution

No resolution available.

Workaround:

You must perform the following steps to recover from NSX-T manager upgrade failure:

1. Login to the NSX-T manager UI and navigate to System > Lifecycle Management > Upgrade and Click Continue to Upgrade.

• If the upgrade process on all the nodes is stuck at 95% (in NSX-T manager UI) then run the following command from NSX-T appliance CLI to verify the upgrade status: get upgrade progress-status
• If the command output displays that ‘Upgrade is in progress’ then restart the install-upgrade service on all the nodes:

           restart service install-upgrade

OR,

Run the following commands on the primary NSX manager node CLI:
Resume upgrade-bundle VMware-NSX-unifies-appliance-3.2.1.2.0.20541216 playbook.
restart service install-upgrade

2. Navigate to System > Configuration > Appliances and verify the following:

• Cluster status is Degraded.  
• Click View Details on each node and verify if the MONITORING service is DOWN on two of the nodes in the Cluster.

If the above holds true then reboot both the node VMs on which the MONITORING service is DOWN by running the command reboot on the NSX-T appliance's CLI. 

3. After the VMs are rebooted, verify that the MONITORING service is UP on the nodes that were rebooted and
NSX-T cluster status is Stable under System > Configuration > Appliances.

4. Retry the failed upgrade from SDDC manager (Updates/Patches tab) and SDDC manager will report that the NSX-T upgrade is successful.

5. You can now run precheck and continue to upgrade vCenter server and ESXi.