ESXi hosts prepared for VMware NSX fail to upgrade during NSX-T upgrade: Failed to load module

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• You are upgrading VMware NSX and the host upgrade is failing.
• In the ESXi logs /var/run/log/esxupdate.log we see the following errors:

LoadKernelModules()   File "/etc/init.d/nsx-datapath-dl", line 217, in LoadKernelModules     nsxesxutils.loadModule(modName, modParam)   File "/usr/lib/vmware/nsx-esx-datapath/lib64/pyth
on/nsxesxutils.py", line 576, in loadModule     raise Exception('Failed to load module %s: %s' % Exception: Failed to load module nsx-esx-70u3/nsxt-ens-21761693: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-ens-21761693): Out of
 memory Cannot load module nsx-esx-70u3/nsxt-ens-21761693: Out of memory   It is not safe to continue. Please reboot the host immediately to discard the unfinished update.. Clean up the installation.

Note: The module nsxt-ens-21761693 in the log above, is just an example and other NSX-T modules may fail with the same message.

• On the ESXi host if we check the size of the driver, we size it is around 50MB:

#vmkload_mod -l | grep -i nvidia
...
nvidia                         13    52372
...

Environment

VMware NSX-T

Cause

The Nvidia driver being used is quite large and consequently the ESXi host is unable to load the new NSX modules into memory and fails to upgrade the host due to this. This occurs in vGPU versions  15.0 to 15.2.
To check which version of the driver is being used, run the following command:

esxcli software component list

Sample result:

NVD-XXXX Driver             525.XXXXXXX

Resolution

There is a newer version of the driver with reduced size. Customers should upgrade to vGPU 15.3 or later.

Workaround:
There is a newer version of the driver with reduced size. Customers should upgrade to vGPU 15.3 or later.