In-place NSX-T ESXi host upgrade failure
Article ID: 324236
Updated On:
Products
VMware NSX
Issue/Introduction
- NSX-T Data Center in-place upgrade
- ESXi host fails to upgrade
- On the ESXi host, /var/log/esxupdate.log shows an out of memory error for nsx-t-vsip module
2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: Cannot load module nsx-esx-70u1/nsxt-vsip-19380480: Out of memory
2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-19380480): Out of memory
2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: Cannot load module nsx-esx-70u1/nsxt-vsip-19380480: Out of memory
2022-04-08T14:37:22Z esxupdate: 124631012: HostImage: DEBUG: installer LiveImageInstaller failed: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: Error in running [/etc/init.d/nsx-datapath-dl start upgrade]: Return code: 1 Output: start upgrade begin Exception: Traceback (most recent call last): File "/etc/init.d/nsx-datapath-dl", line 1154, in <module> DualLoadUpgrade() File "/etc/init.d/nsx-datapath-dl", line 979, in DualLoadUpgrade LoadKernelModules() File "/etc/init.d/nsx-datapath-dl", line 209, in LoadKernelModules nsxesxutils.loadModule(modName, modParam) File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 462, in loadModule (moduleName, out.decode())) Exception: Failed to load module nsx-esx-70u1/nsxt-vsip-19380480: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-19380480): Out of memory
2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-19380480): Out of memory
2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: Cannot load module nsx-esx-70u1/nsxt-vsip-19380480: Out of memory
2022-04-08T14:37:22Z esxupdate: 124631012: HostImage: DEBUG: installer LiveImageInstaller failed: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: Error in running [/etc/init.d/nsx-datapath-dl start upgrade]: Return code: 1 Output: start upgrade begin Exception: Traceback (most recent call last): File "/etc/init.d/nsx-datapath-dl", line 1154, in <module> DualLoadUpgrade() File "/etc/init.d/nsx-datapath-dl", line 979, in DualLoadUpgrade LoadKernelModules() File "/etc/init.d/nsx-datapath-dl", line 209, in LoadKernelModules nsxesxutils.loadModule(modName, modParam) File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 462, in loadModule (moduleName, out.decode())) Exception: Failed to load module nsx-esx-70u1/nsxt-vsip-19380480: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-19380480): Out of memory
NOTE: The module in the "Cannot load module" component may be different. The key issue for this symptom is the "Out of memory" message.
Environment
VMware NSX 3.x
VMware NSX 4.x
VMware NSX 4.x
Cause
- In-place upgrade of ESXi hosts allows hosts to be upgraded without entering maintenance mode.
- Since the ESXi host has to continue running, the current NSX version while loading the upgraded version, it requires more memory than a standard upgrade.
- During this dual load process for the the NSX module, if the ESXi host cannot allocate sufficient memory the upgrade will fail.
Resolution
This is a known issue impacting NSX-T Data Center.
Workaround:
If a host upgrade failure has already occurred, follow these steps to recover.
1. On the ESXi host that has failed to upgrade, clear the upgrade status flag
#nsxcli -c set host-switch upgrade-status false
2. Reboot the host and resume the upgrade
If you are using the "#nsxcli -c set host-switch upgrade-status false" command and encountering the "Internal Error" issue, please open a support case with Broadcom Support.
Feedback
Yes
No
Powered by
