Controller upgrade on some or all nodes before operating system change fails in NSX-T

Products

• VMware NSX

Issue/Introduction

This article provides information on recovery steps when Controller upgrade on some or or all nodes before operating system change fails.

Symptoms:

• Upgrade Coordinator User Interface (UI) displays some or all nodes have failed the upgrade.
• You see error similar to:
  
  Error: [CCP UCP] Install OS failed for CCP node ######## install_os execution failed with msg: MAIN UPGRADE SCRIPT FAILURE REASON: An unexpected exception occurred: Exception
  
  
   

Environment

VMware NSX-T Data Center
VMware NSX-T
VMware NSX-T Data Center 2.x

Cause

This issue occurs because the upgrade has paused. This occurs if any upgrade steps fails to execute, which can in turn be due to multiple system-level issues.

Resolution

To resolve this issue:

1. Identify the Controller node(s) which failed at a given step.
2. Check the syslog on the failed node(s) and on the NSX Manager to identify the cause of the failed step and resolve it.
3. Restart UC.
   a. From the NSX Manager root shell, run the /etc/init.d/upgrade-coordinator restart command.
   b. Restart the upgrade from the NSX User Interface. Note: The MUB will already be available. Use the same MUB.
    
4. As an alternative to step 3, manually run all the remaining upgrade steps from the Controller CLI. After this completes, restart UC to detect that the NSX Controller upgrade has completed.
   
   Note: If recovering AFTER the operating system has changed, follow the documented procedure for rollback.