In-place NSX-T ESXi host upgrade failure

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• NSX-T Data Center in-place upgrade
• ESXi host fails to upgrade
• On the ESXi host, /var/log/esxupdate.log shows an out of memory error for nsx-t-vsip module

2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: Cannot load module nsx-esx-70u1/nsxt-vsip-19380480: Out of memory
2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-19380480): Out of memory
2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: Cannot load module nsx-esx-70u1/nsxt-vsip-19380480: Out of memory
2022-04-08T14:37:22Z esxupdate: 124631012: HostImage: DEBUG: installer LiveImageInstaller failed: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: Error in running [/etc/init.d/nsx-datapath-dl start upgrade]: Return code: 1 Output: start upgrade begin Exception: Traceback (most recent call last):   File "/etc/init.d/nsx-datapath-dl", line 1154, in <module>     DualLoadUpgrade()   File "/etc/init.d/nsx-datapath-dl", line 979, in DualLoadUpgrade     LoadKernelModules()   File "/etc/init.d/nsx-datapath-dl", line 209, in LoadKernelModules     nsxesxutils.loadModule(modName, modParam)   File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 462, in loadModule     (moduleName, out.decode())) Exception: Failed to load module nsx-esx-70u1/nsxt-vsip-19380480: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-19380480): Out of memory

Environment

VMware NSX-T Data Center 3.x
VMware NSX-T Data Center

Cause

In-place upgrade of ESXi hosts allows hosts to be upgraded without the necessity of entering maintenance mode.
Since the ESXi host has to continue running the current NSX version while loading the upgraded version, it requires more memory than a standard upgrade.
During this dual load process for the the DFW vsip module, if the ESXi host cannot allocate sufficient memory the upgrade will fail.

Resolution

This is a known issue impacting NSX-T Data Center.

Workaround:

If a host upgrade failure has already occurred, follow these steps to recover.

      1. On the ESXi host that has failed to upgrade, clear the upgrade status flag

#nsxcli -c set host-switch upgrade-status false

      2. Reboot the host and resume the upgrade