Recovering from an NSX-T In-place Upgrade Failure

Products

• VMware NSX

Issue/Introduction

• NSX-T is being upgraded.
• NSX-T version is 3.0 or above.
• ESXi version is 7.0 or above.
• In-place upgrade of the ESXi hosts is selected.
• The host unit upgrade fails with the following error displayed:

Unexpected error while upgrading upgrade unit: Install of offline bundle failed on host <host-uuid> with error : [LiveInstallationError] ...
It is not safe to continue. Please reboot the host immediately to discard the unfinished update. Please refer to the log file for more details..

• Similar logs as displayed in the UI can also be found on the NSX-T manager logs  /var/log/upgrade-coordinator/upgrade-coordinator.log and ESXi logs of the impacted host /var/run/log/esxupdate.log

Environment

VMware NSX-T Data Center
VMware NSX

Cause

An NSX-T In-place upgrade of an ESXi host failure, can be caused by many things.

 

Some of the most common causes specific to in-place upgrade include: Insufficient memory or failure during dual load of kernel modules.

Resolution

This is a known issue impacting NSX-T Data Center.

Workaround:

1. For an ESXi 7.0 host, enter the nsxcli by running:
   # nsxcli
2. Then use the following command to clear the upgrade status flag on the host:
# set host-switch upgrade-status false
3. Log in to vCenter Server and place the impacted host into maintenance mode.
4. Reboot the host.
5. Exit NSX maintenance mode on the NSX Hosts page.
6. Resume the upgrade process, which will now NOT be an in-place upgrade.
   For detailed documentation on troubleshooting upgrade failures, refer to the following guide: Troubleshooting Upgrade Failures
   For detailed documentation on in-place upgrade failures, refer to the following guide: In-Place Upgrade Fails
   
   

Note: If the below error is encountered when trying to put the host in maintenance mode, it implies the step 2 above was not run or the command did not apply correctly:

"Currently connected network interface" 'Network adapter X' cannot use network ", because "the network is under maintenance".

Note: X denotes the number which be indicated for the adapter and can differ.