In-place NSX-T ESXi host upgrade failure

Products

• VMware NSX

Issue/Introduction

• NSX-T Data Center in-place upgrade
• ESXi host fails to upgrade
• On the ESXi host, /var/log/esxupdate.log shows an out of memory error for nsx-t-vsip module

2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: Cannot load module nsx-esx-70u1/nsxt-vsip-19380480: Out of memory
2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-19380480): Out of memory
2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: Cannot load module nsx-esx-70u1/nsxt-vsip-19380480: Out of memory
2022-04-08T14:37:22Z esxupdate: 124631012: HostImage: DEBUG: installer LiveImageInstaller failed: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: Error in running [/etc/init.d/nsx-datapath-dl start upgrade]: Return code: 1 Output: start upgrade begin Exception: Traceback (most recent call last):   File "/etc/init.d/nsx-datapath-dl", line 1154, in <module>     DualLoadUpgrade()   File "/etc/init.d/nsx-datapath-dl", line 979, in DualLoadUpgrade     LoadKernelModules()   File "/etc/init.d/nsx-datapath-dl", line 209, in LoadKernelModules     nsxesxutils.loadModule(modName, modParam)   File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 462, in loadModule     (moduleName, out.decode())) Exception: Failed to load module nsx-esx-70u1/nsxt-vsip-19380480: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-19380480): Out of memory

 

NOTE: The module in the "Cannot load module" component may be different. The key issue for this symptom is the "Out of memory" message.

Environment

VMware NSX-T 3.x
VMware NSX 4.x

Cause

• In-place upgrade of ESXi hosts allows hosts to be upgraded without entering maintenance mode.
• Since the ESXi host has to continue running, the current NSX version while loading the upgraded version, it requires more memory than a standard upgrade.
• During this dual load process for the the NSX module, if the ESXi host cannot allocate sufficient memory the upgrade will fail.

Resolution

This is a known issue impacting NSX-T Data Center.

Workaround:

If a host upgrade failure has already occurred, follow these steps to recover.

      1. On the ESXi host that has failed to upgrade, clear the upgrade status flag

nsxcli -c set host-switch upgrade-status false

      2. Reboot the host and resume the upgrade

If you are using the "nsxcli -c set host-switch upgrade-status false" command and encountering the "Internal Error" issue, please open a support case with Broadcom Support.

Additional Information

This may also lead to a situation where VMs cannot be vMotioned to or from an impacted host as described in KB -  After NSX-T In-place upgrade failure, vMotions off host fail because "the network is under maintenance"