Failes to uninstall NSX from ESXi with the error "Consumed resource count of module is not zero (bad0145)"

Products

• VMware NSX

Issue/Introduction

• Fails to uninstall NSX from ESXi
• ESXi become "orphaned" state in NSX Manager UI
• The following errors can be found in esxupdate.log
  

  [timestamp] esxupdate: 2127121: root: ERROR: Return code: 1
  [timestamp] esxupdate: 2127121: root: ERROR: Output: stop remove begin
  [timestamp] esxupdate: 2127121: root: ERROR: Exception:
  [timestamp] esxupdate: 2127121: root: ERROR: Traceback (most recent call last):
  [timestamp] esxupdate: 2127121: root: ERROR:   File "/etc/init.d/nsx-datapath", line 2027, in <module>
  [timestamp] esxupdate: 2127121: root: ERROR:     UnloadKernelModules(False, True)
  [timestamp] esxupdate: 2127121: root: ERROR:   File "/etc/init.d/nsx-datapath", line 1710, in UnloadKernelModules
  [timestamp] esxupdate: 2127121: root: ERROR:     unloadModule(modName, 'nsxt-vsip' in modName)
  [timestamp] esxupdate: 2127121: root: ERROR:   File "/etc/init.d/nsx-datapath", line 1666, in unloadModule
  [timestamp] esxupdate: 2127121: root: ERROR:     raise KernelModulesException('Failed to unload module %s: %s' %
  [timestamp] esxupdate: 2127121: root: ERROR: KernelModulesException: Failed to unload module nsxt-ens-22224315: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-ens-22224315 failed: Consumed resource count of module is not zero (bad0145)
  [timestamp] esxupdate: 2127121: root: ERROR: Cannot remove module nsxt-ens-22224315: Consumed resource count of module is not zero
  [timestamp] esxupdate: 2127121: root: ERROR:
  [timestamp] esxupdate: 2127121: root: ERROR:
  [timestamp] esxupdate: 2127121: root: ERROR: It is not safe to continue. Please reboot the host immediately to discard the unfinished update.

Environment

VMware NSX

Cause

Based on  the logs above, nsxt-ens-22224315 unload was delayed because its heap was in use.
kernel module stays busy and cannot be unloaded until user world configuration is active and its not in kernel module's control when user world clears the resource.
A kernel module unload may return 'mod in use' or busy if it has references.

Resolution

The workaround is to reboot the ESXi host and then retry to uninstall using the nsx del command.

Uninstall NSX from a vSphere Cluster
https://techdocs.broadcom.com/us/en/vmware-cis/nsx/vmware-nsx/4-2/installation-guide/uninstall-nsx-from-a-host-transport-node/remove-nsx-from-a-cluster.html

And the issue is fixed in NSX 4.2.0.
All kernel modules will be retried for unloading if a failure occurs.