NSX Manager upgrade does not proceed after 2 Manager upgrade completed

Products

• VMware NSX

Issue/Introduction

• You are upgrading NSX Manager and upgrade-from version is 3.2.1.x or later.
  
  • It is rolling upgrade.
• 2 Manager nodes has been upgraded successfully.
• The last node started upgrade but it's in progress forever at 1%.
• In [System] - [Backup and Restore], Upgrade history is loading forever.
  
  • Check all the Manager nodes.

Environment

VMware NSX-T Datacenter 3.2

VMware NSX 4.0
VMware NSX 4.1

Cause

Upgrade process is handed over from the orchestrator node to another Manager node when the last Manager node is upgraded.

The hand-over gets stuck forever if the Manager node that the upgrade process is handed over to can't retrieve backup files on the backup server.

Resolution

The issue won't be seen when upgrading to NSX 4.2 or later.

If upgrade gets stuck, follow the steps to resume upgrade.

1. Make sure all the NSX Manager can list backup.
   • Make sure you can log in the backup server with the configured credential.
     The KB might be helpful.
     NSX-T Manager Backup task never completes
     https://knowledge.broadcom.com/external/article?articleNumber=371358
2. Restart Upgrade Coordinator on all the 3 Manager nodes.
   You can restart the service in NSX CLI restart service install-upgrade
3. Now you can retry upgrade.