NSX-T Host upgrade Fails in mixed switch type environment (nsx vds+ nsx on dvpg)

Products

• VMware NSX

Issue/Introduction

Symptoms:

- You are preforming a VMware NSX upgrade to 4.1.2

- Checking "net-dvs -l" output there should be a switch with vswitch type and another switch with cswitch type with kcp property set.

- Host upgrade failed with the below error:

File "/etc/init.d/nsx-datapath", line 1592, in <module>     UnloadKernelModules(False, True)   File "/etc/init.d/nsx-datapath", line 1331, in UnloadKernelModules     unloadModule(modName, 'nsxt-vsip' in modName)   File "/etc/init.d/nsx-datapath", line 1303, in unloadModule     raise KernelModulesException('Failed to unload module %s: %s' % KernelModulesException: Failed to unload module nsxt-kcp-22104635: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-kcp-22104635 failed: Busy (bad0004) Cannot remove module nsxt-kcp-22104635:

- Disabling the kcp property will not work:

       Example:
       [root@esx:~] net-dvs -u com.vmware.nsx.kcp.enable -p hostPropList DVS1   

- Fails to unload kcp because kcp property of dvpg switch is not removed.

- Running the command "del nsx" in nsxcli mode would not help clean up the VIBs

- Relevant log’s location : /var/log/nsx-syslog.log

Environment

VMware NSX 4.1.2.x

Cause

- KernelModulesException: Failed to unload module nsxt-kcp-22104635: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-kcp-22104635 failed: Busy (bad0004)

 

Resolution

Fix is available in future releases.

Workaround:

1) Proceed with in-place host upgrade

2) If option 1 did not work then run the below command on each ESXi host *before* the upgrade :

# mkdir /tmp/nsx2; echo '{"dual_load":true}' > /tmp/nsx2/debug