NSX VIBs are not removed from an ESXi host

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

Trying to remove NSX VIB from ESXi host(s) fails with error message displayed on screen and ESXi log bundle /var/log/esxupdate.log:

Unable to remove vib on host. java.rmi.RemoteException: [LiveInstallationError] VMware_bootbank_nsx-esx-datapath_4.1.0.0.0-7.0.21236273: Error in running [/etc/init.d/nsx-datapath stop remove]: Return code: 1 Output: stop remove begin Exception:Traceback (most recent call last): File "/etc/init.d/nsx-datapath", line 1995, in <module> UnloadKernelModules(False, True) File "/etc/init.d/nsx-datapath", line 1681, in UnloadKernelModules unloadModule(modName, 'nsxt-vsip' in modName) File "/etc/init.d/nsx-datapath", line 1638, in unloadModule (moduleName, out.decode())) KernelModulesException: Failed to unload module nsxt-kcp-21####73: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-kcp-21236273 failed: Busy (bad0004) Cannot remove module nsxt-kcp-21236273"

The above unloading error may also be seen for other modules such as:

Unloading module nsxt-sfp-21####73 failed: Busy (bad0004) Cannot remove module nsxt-kcp-21####73

Unloading module nsxt-vdl2-21####73 failed: Busy (bad0004) Cannot remove module nsxt-kcp-21####73

Running 'del nsx' on the ESXi host in the nsxcli shell does not remove the VIB's.
In the NSX-T UI removing NSX-T using force, also fails to remove the VIB's.
Reboot of the host does not resolve the issue.
Trying to prepare ESXi hosts that had NSX-T removed using force option in the NSX-T UI fails.

On the ESXi you see one or more of the three modules noted below are loaded and enabled:

net-dvs -l | grep com.vmware.nsx
com.vmware.nsx.kcp.enable = true ,   propType = CONFIG
com.vmware.nsx.spf.enabled = true ,  propType = CONFIG
com.vmware.nsx.vdl2.enabled = true , propType = CONFIG

Environment

VMware NSX
VMware NSX-T Data Center

Cause

The issue is caused when one or more of the modules com.vmware.nsx.spf, com.vmware.nsx.vdl2, com.vmware.nsx.kcp are loaded and enabled and preventing the NSX VIB's from being removed.

Resolution

This is a known issue impacting VMware NSX and VMware NSX-T Data Center.

Workaround:

Confirm using the following command that the module displayed in the error is still enabled:
```
net-dvs -l | grep com.vmware.nsx
```
Note: You will see output similar to the following:
```
com.vmware.nsx.kcp.enable = true ,   propType = CONFIG
com.vmware.nsx.spf.enabled = true ,  propType = CONFIG
com.vmware.nsx.vdl2.enabled = true , propType = CONFIG
```
Note: Any entries without CONFIG at the end can be ignored.

Obtain the name of the vDS which needs to be modified via the esxcfg-vswitch -l command.

Note: You will see output similar to the following:

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0         8960        1           128               1500

  PortGroup Name                            VLAN ID  Used Ports  Uplinks
  VM Network                                0        0

Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch1         8960        5           128               1500    vmnic2,vmnic3

  PortGroup Name                            VLAN ID  Used Ports  Uplinks
  PG_1                                      1111     0           vmnic2,vmnic3
  PG_2                                      1112     0           vmnic2,vmnic3
  PG_3                                      1113     0           vmnic2,vmnic3  

DVS Name         Num Ports   Used Ports  Configured Ports  MTU     Uplinks
NSX_VDS          8960         58         1024              9000    vmnic8,vmnic9

  DVPort ID                               In Use      Client
  40                                      1           vmnic2
  52                                      1           vmnic3
  1                                       1           vmk1
  7                                       1           vmk2

Note: NSX_VDS is the name of the vDS to be modified. There might be more than one vDS that needs to be modified.

For each vDS in use by NSX, issue a command similar to the following to disable the active NSX kernel module(s):
```
net-dvs -u com.vmware.nsx.kcp.enable -p hostPropList NSX_VDS
```
Repeat Step 3 for any other active NSX kernel modules found in Step 1 on each remaining vDS in use by NSX.
Repeat Step 1 to validate that the NSX kernel modules have been disabled.

Note: If the NSX kernel modules are not disabled, reboot the host and repeat Steps 3 and 4.
If the host is not already in maintenance mode, put it into maintenance mode before proceeding to Step 7.
Issue the nsxcli -c del nsx command to remove the NSX VIBs.
Issue the esxcli software vib list | grep -E 'nsx|vsipfwlib' command to validate that no NSX VIBs are installed on the host. There should be no output.
If the ESXi host still has an error in the NSX UI, perform the removal again in the NSX UI, using the force delete option.

Additional Information

If the host is rebooted, you may see the NSX kernel modules activated again. This is expected behavior and should not affect the host once the NSX VIBs are removed.