Recovering from an NSX-T In-place Upgrade Failure

Products

• VMware NSX
• VMware NSX-T Data Center

Issue/Introduction

• You are upgrading NSX-T.
• NSX-T version is 3.0 or above.
• ESXi version is 7.0 or above.
• You are doing an in-place upgrade of the ESXi hosts.
• The host unit upgrade fails with the following error displayed:

Unexpected error while upgrading upgrade unit: Install of offline bundle failed on host ,host-uuid> with error : [LiveInstallationError] ...
It is not safe to continue. Please reboot the host immediately to discard the unfinished update. Please refer to the log file for more details..

• Similar logs as displayed in the UI can also be found on the NSX-T manager logs  /var/log/upgrade-coordinator/upgrade-coordinator.log and ESXi logs of the impacted host /var/run/log/esxupdate.log. 

Environment

VMware NSX-T Data Center
VMware NSX

Cause

An NSX-T In-place upgrade of an ESXi host failure, can be caused by many things.
Some of the most common causes specific to in-place upgrade include: insufficient memory or failure during dual load of kernel modules.

Resolution

This is a known issue impacting NSX-T Data Center.

Workaround:

1. For an ESXi 7.0 host, enter the nsxcli by running:

# nsxcli

2. Then use the following command to clear the upgrade status flag on the host:

> set host-switch upgrade-status false

3. Log in to vCenter Server and place the impacted host into maintenance mode.

4. Reboot the host.

5. Exit NSX maintenance mode on the NSX Hosts page.

6. Resume the upgrade process, which will now NOT be an in-place upgrade.

For detailed documentation on Troubleshooting Upgrade Failures, refer to the following guide:
Troubleshooting Upgrade Failures

For detailed documentation on in-place upgrade failure, refer to the following guide:
In-Place Upgrade Fails

Note: If we encounter the below error, when trying to put the host in maintenance mode, it implies the step 2 above was not run or the command did not apply correctly:

"Currently connected network interface" 'Network adapter X' cannot use network ", because "the network is under maintenance".

Note: X denoted the number which be indicated for the adapter and can differ.