NSX-T fail to remove NSX-T VIB's from ESXi host: Busy (bad0004) Cannot remove module nsxt-kcp, nsxt-sfp, nsxt-vdl2

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• Version of NSX-T in use 3.2.x and 4.x.
• Trying to remove NSX-T VIB from ESXi host(s) fails with error message displayed on screen and ESXi log bundle /var/log/esxupdate.log:

Unable to remove vib on host. java.rmi.RemoteException: [LiveInstallationError] VMware_bootbank_nsx-esx-datapath_4.1.0.0.0-7.0.21236273: Error in running [/etc/init.d/nsx-datapath stop remove]: Return code: 1 Output: stop remove begin Exception:Traceback (most recent call last): File "/etc/init.d/nsx-datapath", line 1995, in <module> UnloadKernelModules(False, True) File "/etc/init.d/nsx-datapath", line 1681, in UnloadKernelModules unloadModule(modName, 'nsxt-vsip' in modName) File "/etc/init.d/nsx-datapath", line 1638, in unloadModule (moduleName, out.decode())) KernelModulesException: Failed to unload module nsxt-kcp-21236273: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-kcp-21236273 failed: Busy (bad0004) Cannot remove module nsxt-kcp-21236273"

• The above unloading error may also be seen for other modules such as:

Unloading module nsxt-sfp-21236273 failed: Busy (bad0004) Cannot remove module nsxt-kcp-21236273

Unloading module nsxt-vdl2-21236273 failed: Busy (bad0004) Cannot remove module nsxt-kcp-21236273

• Running 'del nsx' on the ESXi host in the nsxcli shell does not remove the VIB's.
• In the NSX-T UI removing NSX-T using force, also fails to remove the VIB's.
• Reboot of the host does not resolve the issue.
• Trying to prepare ESXi hosts that had NSX-T removed using force option in the NSX-T UI fails.
• On the ESXi we see one or more of the three modules are loaded and enabled:

net-dvs -l | grep com.vmware.nsx.kcp
                com.vmware.nsx.kcp.enable = true , propType = CONFIG
                com.vmware.nsx.spf.enable = true , propType = CONFIG
                com.vmware.nsx.vdl2.enable = true , propType = CONFIG

Environment

VMware NSX-T Data Center
VMware NSX-T Data Center 4.x
VMware NSX-T Data Center 3.x

Cause

The issue is caused when one or more of the modules com.vmware.nsx.spf, com.vmware.nsx.vdl2, com.vmware.nsx.kcp is loaded and enable and preventing the NSX-T VIB's from being removed.

Resolution

This is a known issue impacting VMware NSX and NSX-T data center.

Workaround:
1. First confirm using the below command that the module displayed in the error is still enabled:

# net-dvs -l | grep com.vmware.nsx.kcp
                com.vmware.nsx.kcp.enable = true , propType = CONFIG
                com.vmware.nsx.kcp.enable = true , propType = CONFIG

2. Then we need to disable the module for each DVS in use, using the following syntax:

# net-dvs -u "<property>" -p hostPropList <switchName>

3. To find the DVS names:

# esxcfg-vswitch -l

4. For example for DVS named RegionA01-VDS7:

# net-dvs -u com.vmware.nsx.kcp.enable -p hostPropList RegionA01-VDS7

5. Using the command from step 1, check the module is disabled.
6. Place the ESXi host in vSphere maintenance mode and on the ESXi nsxcli shell run:

# nsxcli> del nsx

7. Confirm the NSX-T VIBs have been remove: esxcli software vib list | grep -i nsx
8. If the ESXi hosts still have some error on the NSX-T UI, perform the removal again in the NSX-T UI, using the force delete option.
9. If you still find the DVS has the com.vmware.nsx.kcp enabled, please reboot the host and repeat step 2.