NSX Uninstallation fails/stuck on ESX host and VIBs cannot be removed.

Products

• VMware NSX

Issue/Introduction

Symptoms:

Uninstall of NSX VIBs on ESXi host through del nsx command fails with the following error in /var/log/nsx-syslog.log:

Unable to remove vib on host. java.rmi.RemoteException: [LiveInstallationError] VMware_bootbank_nsx-esx-datapath_4.1.0.0.0-7.0.21236273:Error in running [/etc/init.d/nsx-datapath stop remove]: Return code: 1 Output: stop remove begin Exception:Traceback (most recent call last): File "/etc/init.d/nsx-datapath", line 1995, in <module> UnloadKernelModules(False, True) File "/etc/init.d/nsx-datapath", line 1681, in UnloadKernelModules unloadModule(modName, 'nsxt-vsip' in modName) File "/etc/init.d/nsx-datapath", line 1638, in unloadModule (moduleName, out.decode())) KernelModulesException: Failed to unload module nsxt-kcp-21236273: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-kcp-21236273 failed: Busy (bad0004) Cannot remove module nsxt-kcp-21236273"

Environment

VMware NSX-T Data Center

VMware NSX

Cause

The kcp module is still enabled while it should be disabled for the module to be properly removed.

Resolution

Workaround:
1. Run the command to confirm kcp module is in 'enable' state and confirm that it is the cause of issue, each line reflect a DVS hitting the issue.
      [root@esx:~] net-dvs -l | grep com.vmware.nsx.kcp
                com.vmware.nsx.kcp.enable = true , propType = CONFIG
                com.vmware.nsx.kcp.enable = true , propType = CONFIG

  2. Issue the following command for each DVS
     Command syntax: net-dvs -u "<property>" -p hostPropList <switchName>

     Example:
       [root@esx:~] net-dvs -u com.vmware.nsx.kcp.enable -p hostPropList DVS1
       [root@esx:~] net-dvs -u com.vmware.nsx.kcp.enable -p hostPropList DVS2

  3. Repeat step 1 to confirm the property is set to disabled for all DVS

  4. Put the ESXi host in Maintenance mode and perform the del nsx in nsxcli mode.

  5. Verify all the NSX VIBs are removed:
       [root@esx:~] esxcli software vib list | grep nsx

  6. If the host still shows an error in UI, force delete the host to clear the UI error.