Failed to load module nsxt-ens observed when performing in-place NSX transport node upgrade.
search cancel

Failed to load module nsxt-ens observed when performing in-place NSX transport node upgrade.

book

Article ID: 409603

calendar_today

Updated On:

Products

VMware NSX

Issue/Introduction

  • While performing an NSX in-place upgrade a host fails to upgrade successfully. 
  • Enhanced Datapath - EDP (formerly called ENS) is configured on impacted transport nodes.
  • NSX Upgrade GUI shows an error similar to the below

    Upgrade failed for <Transport Node ID> host. Please reboot the host immediately to discard the unfinished update.

    Unexpected error while upgrading upgrade unit: Install of offline bundle failed on host <TN ID> with error : [LiveInstallationError] VMware_bootbank_nsx-esx-datapath_4.2.2.1.0-8.0.24765085: Error in running [/etc/init.d/nsx-datapath-dl start upgrade]: Return code: 1 Output: start upgrade begin Exception: Traceback (most recent call last): File "/etc/init.d/nsx-datapath-dl", line 1354, in <module> DualLoadUpgrade() File "/etc/init.d/nsx-datapath-dl", line 1152, in DualLoadUpgrade LoadKernelModules() File "/etc/init.d/nsx-datapath-dl", line 218, in LoadKernelModules nsxesxutils.loadModule(modName, modParam) File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 607, in loadModule raise Exception('Failed to load module %s: %s' % Exception: Failed to load module nsx-esx-80u3/nsxt-ens-24765085: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-ens-24765085): Failure Cannot load module nsx-esx-80u3/nsxt-ens-24765085: Failure It is not safe to continue. Please reboot the host immediately to discard the unfinished update. cause = ('nsx-lcp-bundle(4.2.2.1.0-8.0.24765085)', 'nsx-datapath-dl', 'Error in running [/etc/init.d/nsx-datapath-dl start upgrade]:\nReturn code: 1\nOutput: start upgrade begin\nException:\nTraceback (most recent call last):\n File "/etc/init.d/nsx-datapath-dl", line 1354, in <module>\n DualLoadUpgrade()\n File "/etc/init.d/nsx-datapath-dl", line 1152, in DualLoadUpgrade\n LoadKernelModules()\n File "/etc/init.d/nsx-datapath-dl", line 218, in LoadKernelModules\n nsxesxutils.loadModule(modName, modParam)\n File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 607, in loadModule\n raise Exception(\'Failed to load module %s: %s\' %\nException: Failed to load module nsx-esx-80u3/nsxt-ens-24765085: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-ens-24765085): Failure\nCannot load module nsx-esx-80u3/nsxt-ens-24765085: Failure\n\n') vibs = ['VMware_bootbank_nsx-esx-datapath_4.2.2.1.0-8.0.24765085'] Please refer to the log file for more details. Please refer 'https://kb.vmware.com/s/article/91383' article for troubleshooting steps.

  • Upgrade Coordinator logs on the NSX Manager will show the same above error in the following format:

    58190:2025-08-29T04:05:30.952Z <NSX-Manager> NSX 1128356 FABRIC [nsx@6876 comp="nsx-manager" errorCode="MP31806" level="ERROR" subcomp="upgrade-coordinator"] Install of offline bundle failed on host <Host IP> with error : java.rmi.RemoteException:  [LiveInstallationError] VMware_bootbank_nsx-esx-datapath_4.2.2.1.0-8.0.24765085: Error in running [/etc/init.d/nsx-datapath-dl start upgrade]: Return code: 1 Output: start upgrade begin Exception: Traceback (most recent call last):   File "/etc/init.d/nsx-datapath-dl", line 1354, in <module>     DualLoadUpgrade()   File "/etc/init.d/nsx-datapath-dl", line 1152, in DualLoadUpgrade     LoadKernelModules()   File "/etc/init.d/nsx-datapath-dl", line 218, in LoadKernelModules     nsxesxutils.loadModule(modName, modParam)   File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 607, in loadModule     raise Exception('Failed to load module %s: %s' % Exception: Failed to load module nsx-esx-80u3/nsxt-ens-24765085: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-ens-24765085): Failure Cannot load module nsx-esx-80u3/nsxt-ens-24765085: Failure   It is not safe to continue. Please reboot the host immediately to discard the unfinished update.      cause = ('nsx-lcp-bundle(4.2.2.1.0-8.0.24765085)', 'nsx-datapath-dl', 'Error in running [/etc/init.d/nsx-datapath-dl start upgrade]:\nReturn code: 1\nOutput: start upgrade begin\nException:\nTraceback (most recent call last):\n  File "/etc/init.d/nsx-datapath-dl", line 1354, in <module>\n    DualLoadUpgrade()\n  File "/etc/init.d/nsx-datapath-dl", line 1152, in DualLoadUpgrade\n    LoadKernelModules()\n  File "/etc/init.d/nsx-datapath-dl", line 218, in LoadKernelModules\n    nsxesxutils.loadModule(modName, modParam)\n  File "/usr/lib/vmware/nsx-esx-datapath/lib64/python/nsxesxutils.py", line 607, in loadModule\n    raise Exception(\'Failed to load module %s: %s\' %\nException: Failed to load module nsx-esx-80u3/nsxt-ens-24765085: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-ens-24765085): Failure\nCannot load module nsx-esx-80u3/nsxt-ens-24765085: Failure\n\n')       vibs = ['VMware_bootbank_nsx-esx-datapath_4.2.2.1.0-8.0.24765085'] Please refer to the log file for more details.

  • ESXi syslog.log shows a failure (build numbers will differ depending on versions involved):

    2025-08-29T03:53:31Z In(14) init.d/nsx-datapath-dl: INFO: nsx-datapath-dl START
    2025-08-29T03:53:31Z In(14) init.d/nsx-datapath-dl: INFO: Dual Loading Upgrade Case
    2025-08-29T03:53:31Z In(14) init.d/nsx-datapath-dl: INFO: FROM BUILD: 23786738 TO BUILD: 24765085
    ...
    2025-08-29T03:53:35.459Z No(13) vmkload_mod[13058530]: Module 'nsxt-ens-24765085' load by uid=0 who=root unsuccessful
    2025-08-29T03:53:35Z Wa(12) init.d/nsx-datapath-dl: WARNING: nsx-datapath-dl upgrade failed: Failed to load module nsx-esx-80u3/nsxt-ens-24765085: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-ens-24765085): Failure
    2025-08-29T03:53:35Z Wa(12)[+] init.d/nsx-datapath-dl: Cannot load module nsx-esx-80u3/nsxt-ens-24765085: Failure

     

  • At the same time ESXi vmkernel.log shows.

    2025-08-29T03:53:35.383Z In(182) vmkernel: cpu80:13058530)Loading module nsxt-ens-24765085 ...
    2025-08-29T03:53:35.450Z Wa(180) vmkwarning: cpu80:13058530)WARNING: CharDriver: 356: Driver with name ens is already using slot 111
    2025-08-29T03:53:35.450Z Wa(180) vmkwarning: cpu80:13058530)WARNING: Failed to create ENS char device
    2025-08-29T03:53:35.451Z In(182) vmkernel: cpu80:13058530)nsxt-ens-24765085 failed to load.

  • Dataplane is not impacted during this failure.

    Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware NSX-T Data Center
VMware NSX 

Cause

In-place upgrade allows the host to not be placed in to maintenance mode, it then loads both told old and new modules and hot swaps between them allowing a seamless upgrade. This issue occurs due to the ENS module still being in use while attempting to be swapped in a live upgrade situation. This leads to the replacement of the module to fail and the in-place upgrade to subsequently fail as well. This will not occur on every host and depends upon environmental variables at the time of the load attempt. Dataplane will not be impacted at time of the failure.

Resolution

This issue is resolved in VMware NSX 9.0, available at Broadcom downloads.

If you are having difficulty finding and downloading software, please review the Download Broadcom products and software KB.

Recovery

Once this issue has occurred the following KB will need to be followed Recovering from an NSX-T In-place Upgrade Failure.This will require a reboot and a maintenance mode upgrade to be performed to complete the impacted hosts upgrades.

Additional Information

This issue will not occur when upgrading to a version containing the fix.

Powered by Wolken Software