NSX VIBs are not removed from an ESXi host

Products

VMware NSX VMware vSphere ESXi

Issue/Introduction

Trying to remove NSX VIB from ESXi host(s) fails with bellow error message displayed in ESXi log /var/log/esxupdate.log

Unable to remove vib on host. java.rmi.RemoteException: [LiveInstallationError] VMware_bootbank_nsx-esx-datapath_4.1.0.0.0-7.0.21236273: Error in running [/etc/init.d/nsx-datapath stop remove]: Return code: 1 Output: stop remove begin Exception:Traceback (most recent call last): File "/etc/init.d/nsx-datapath", line 1995, in <module> UnloadKernelModules(False, True) File "/etc/init.d/nsx-datapath", line 1681, in UnloadKernelModules unloadModule(modName, 'nsxt-vsip' in modName) File "/etc/init.d/nsx-datapath", line 1638, in unloadModule (moduleName, out.decode())) KernelModulesException: Failed to unload module nsxt-kcp-21####73: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-kcp-21236273 failed: Busy (bad0004) Cannot remove module nsxt-kcp-21236273"

The above unloading error may also be seen for other modules such as:

Unloading module nsxt-sfp-21####73 failed: Busy (bad0004) Cannot remove module nsxt-kcp-21####73

Unloading module nsxt-vdl2-21####73 failed: Busy (bad0004) Cannot remove module nsxt-kcp-21####73

Running 'del nsx' on the ESXi host in the nsxcli shell does not remove the VIB's.
In the NSX-T UI removing NSX-T using force, also fails to remove the VIB's.
Reboot of the host does not resolve the issue.
Trying to prepare ESXi hosts that had NSX-T removed using force option in the NSX-T UI fails.
On the ESXi you see one or more of the three modules noted below are loaded and enabled:

net-dvs -l | grep com.vmware.nsx
com.vmware.nsx.kcp.enable = true ,   propType = CONFIG
com.vmware.nsx.spf.enabled = true ,  propType = CONFIG
com.vmware.nsx.vdl2.enabled = true , propType = CONFIG

Removing proxy NSX dvs from the host was giving below error.

An error occurred during host configuration Operation failed, diagnostics report DVPort "vdrPort" on DVSwitch "DvsPortset-1" is still in use vdr-vdrPort

Environment

VMware NSX
VMware NSX-T Data Center

Cause

The issue is caused when one or more of the modules com.vmware.nsx.spf, com.vmware.nsx.vdl2, com.vmware.nsx.kcp are loaded and enabled and preventing the NSX VIB's from being removed.

Resolution

This is a known issue impacting VMware NSX

Update to vSphere 8.0.3 P08 or later to resolve this issue.

If upgrade is not possible, please consider below workarounds:

Scan environment

To scan and clean your vCenter Server Appliance (VCSA) / ESXi hosts, please follow these steps.

Transfer the Script copy the nsx_cleanup.py script to the VCSA appliance. Place the file in either the /tmp or /root directory.
Execute the Cleanup Script Run the Python script
```
python nsx_cleanup.py
```
Perform a Read-Only Scan by selcting Option 1. This will scan environment in read-only mode to determine if any cleanup is required.
Execute the cleanup only after reviewing the read-only scan results. Then proceed with running the script in cleanup mode against the VC or ESXi hosts.

Cleanup environment (Steps to run from VCSA)

vCenter :-

Follow these steps to perform the cleanup operation on the vCenter Server Appliance (VCSA)

1. Stop the VPXD Service Before executing the database cleanup, you must stop the vCenter Server service
```
service-control --stop vmware-vpxd
```
2. Backup the VCDB Critical: Always take a backup of the vCenter Database (VCDB) before running the cleanup operation. Run the following command to create the backup in the /var/tmp directory:
```
/opt/vmware/vpostgres/current/bin/pg_dump -U postgres -d VCDB -F c -f /var/tmp/vcdb_backup.dump
```
Alternative Backup Path: If you prefer or need to back up to the core storage, you can use the following command instead (verify storage space before proceeding):
```
/opt/vmware/vpostgres/current/bin/pg_dump -d VCDB -U postgres -Fp -f /storage/core/VCDBackUp.bak
```
3. Execute the Cleanup Script Run the attached Python script "python nsx_cleanup.py" to perform the database cleanup.
- When prompted, select Option 2 (Cleanup Environment).
- Note: This is the same script used for both vCenter and ESXi host cleanups.
4. Start the VPXD Service Once the cleanup is complete and the script has successfully exited, restart the vCenter Server service to bring the environment back online:
```
service-control --start vmware-vpxd
```

ESXi :-

Follow these steps to perform the cleanup operation on your ESXi hosts:

1. Create the Host Configuration File In the same directory where your nsx_cleanup.py script is located in your VCSA appliance, you need to create a JSON file named esxi_hosts.json. You can do this using a text editor like vi
```
vi esxi_hosts.json
```
Populate the file with your ESXi host IPs and passwords using the following format:
```
[
    {
        "ip": "10.xxx.199.113",
        "password": "XXYY"
    },
    {
        "ip": "10.xxx.199.114",
        "password": "XXYY"
    }
]
```
2. There are no services on the ESXi hosts to stop before running nsx_cleanup.py script.

3. Enable SSH on all hosts which are being scanned

4. Execute the Cleanup Script Run the Python script, making sure to append the --host-file flag pointing to the JSON file you just created:
```
python nsx_cleanup.py --host-file esxi_hosts.json
```
5. Select the Cleanup Option When the script prompts you, select Option 2 (Cleanup Environment). (Note: This is the exact same script used for the vCenter cleanup).

6. Completion Allow the script to run. Once the cleanup process is finished, the script will exit automatically.

If the above step do not resolve the issue Raise a support ticket with Broadcom support selecting NSX as the product.

Handling Log Bundles for offline review with Broadcom support.

Additional Information

If the host is rebooted, you may see the NSX kernel modules activated again. This is expected behavior and should not affect the host once the NSX VIBs are removed.

Attachments

nsx_cleanup.py get_app