ESXi hosts prepared for VMware NSX fail to upgrade during NSX-T upgrade: Failed to load module

Products

• VMware NSX

Issue/Introduction

• You are upgrading VMware NSX and the host upgrade is failing.
• In the ESXi logs /var/run/log/esxupdate.log we see the following errors:

     LoadKernelModules()   File "/etc/init.d/nsx-datapath-dl", line 217, in LoadKernelModules     nsxesxutils.loadModule(modName, modParam)   File "/usr/lib/vmware/nsx-esx-datapath/lib64/pyth
     on/nsxesxutils.py", line 576, in loadModule     raise Exception('Failed to load module %s: %s' % Exception: Failed to load module nsx-esx-70u3/nsxt-ens-21761693: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-ens-21761693): Out of
     memory Cannot load module nsx-esx-70u3/nsxt-ens-21761693: Out of memory   It is not safe to continue. Please reboot the host immediately to discard the unfinished update.. Clean up the installation.

Note: The module nsxt-ens-21761693 in the log above, is just an example and other NSX-T modules may fail with the same message.

• On the ESXi host if we check the size of the driver, we size it is around 50MB:

          #vmkload_mod -l | grep -i nvidia
...
          nvidia                         13    52372
...

Environment

VMware NSX

Cause

The Nvidia driver being used is quite large and consequently the ESXi host is unable to load the new NSX modules into memory and fails to upgrade the host due to this. This occurs in vGPU versions  15.0 to 15.2.
To check which version of the driver is being used, run the following command:

 

esxcli software component list

 

Sample result:

   NVD-XXXX Driver           525.XXXXXXX

Resolution

Workaround:
There is a newer version of the driver with reduced size. Customers should upgrade to vGPU 15.3 or later.