ESXi hosts prepared for VMware NSX fail to upgrade during NSX-T upgrade: Failed to load module

Products

• VMware NSX

Issue/Introduction

• You are upgrading VMware NSX, and the host upgrade is failing.
• In the ESXi logs /var/run/log/esxupdate.log we see the following errors:

 LoadKernelModules()   File "/etc/init.d/nsx-datapath-dl", line 217, in LoadKernelModules     nsxesxutils.loadModule(modName, modParam)   File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 576, in loadModule     raise Exception('Failed to load module %s: %s' % Exception: Failed to load module nsx-esx-70u3/nsxt-ens-21761693: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-ens-21761693): Out of memory Cannot load module nsx-esx-70u3/nsxt-ens-21761693: Out of memory   It is not safe to continue. Please reboot the host immediately to discard the unfinished update.. Clean up the installation.

Note: The module nsxt-ens-21761693 in the log above is just an example, and other NSX-T modules may fail with the same message.

• On the ESXi host if we check the size of the driver, we find that it is around 50MB:

          #vmkload_mod -l | grep -i nvidia
       nvidia      13    52372

Environment

VMware NSX

Cause

The Nvidia driver being used is quite large in size, and consequently, the ESXi host is unable to load the new NSX modules into memory, failing to upgrade the host as a result. This occurs in vGPU versions 15.0 to 15.2.

To check which version of the driver is being used, run the following command:

 

esxcli software component list

 

Sample result:

   NVD-XXXX Driver           525.XXXXXXX

Resolution

Workaround:
There is a newer version of the driver with a reduced size. Customers should upgrade to vGPU 15.3 or later.