Deleting NSX-T VIB from an ESXi host fails using del nsx: Unable to clear DVS propertycom.vmware.nsx.vdl2.enabled: Status(bad0004)= Busy

Products

• VMware NSX Networking

Issue/Introduction

Symptoms:

• An ESXi host is in an orphaned state in the NSX-T manager.
• In an attempt to clean it up and rebuilt it, del nsx command was run in the nsxcli of the host and failed.
• On the ESXi logs /var/run/log/nsxaVim.log we see the following log entry:

nsxaVim: [2102769]: ERROR ApplyNsxDvsConfig on (50 0d a0 72 e3 8d 15 44-c3 e2 bd a7 a5 e0 5d 98) failed: {'fault': 'PlatformConfigFault', 'msg': 'An error occurred during host configuration.', 'faultMessage': ['Operation failed, diagnostics report: Unable to clear DVS propertycom.vmware.nsx.vdl2.enabled: Status(bad0004)= Busy']}^@
nsxaVim: [2102769]: ERROR Failed to hot-swap cvds ['50 0d a0 72 e3 8d 15 44-c3 e2 bd a7 a5 e0 5d 98']^@
nsxaVim: [2102769]: INFO HotSwapCvds result: Failed to hot-swap cvds ['50 0d a0 72 e3 8d 15 44-c3 e2 bd a7 a5 e0 5d 98']ApplyNsxDvsConfig on (50 0d a0 72 e3 8d 15 44-c3 e2 bd a7 a5 e0 5d 98) failed: {'fault': 'PlatformConfigFault', 'msg': 'An error occurred during host configuration.', 'faultMessage': ['Operation failed, diagnostics report: Unable to clear DVS propertycom.vmware.nsx.vdl2.enabled: Status(bad0004)= Busy']}^@

...
nsxaVim: [2102769]: INFO Invoking ApplyDvsList: dvsParams: ;reset=True;management;deleteTn=True | preDvsList: ['']
nsxaVim: [2102769]: INFO dvs to reset [[]]

Environment

VMware NSX-T

Cause

This issue occurs when the list of DVS's to be cleared are empty, as seen in the log entry above (INFO dvs to reset [[]]) and del nsx is unable to remove NSX-T from the host.

Resolution

This is a known issue impacting VMware NSX.

Workaround:

• First, manually remove the VTEP vmkernel interfaces on the ESXi host:

esxcli network ip interface remove --interface-name=vmk10 - repeat for all VTEP vmkernel interfaces.

• Then, manually remove the VTEP, vdrPort and spfPort ports from the DVS:

net-dvs [ -A | -D ] -p port switch_name - add/delete a port from a dvswitch

• Sample command:

net-dvs -D -p xxxxxxxx-9801-4fed-xxxxxxxxxxxx dvs-name

• To find the port used for the VTEP/vdrPort/spfPort , run the command:

net-dvs -l -> then search the results for VTEP and the port should be listed just above the word VTEP, i.e. if you have 2 VTEP's, you should see the entry twice and need to run the command twice to remove both VTEP ports from the DVS.

• Run del nsx again on the ESXi host.
• If this still fails, you can run the following command, to force the hot swap of the CVDS to DVS:

./usr/lib64/vmware/nsx-opsagent/bin/nsxcfg-vswitch -s

• Then rerun del nsx again to complete the removal of NSX-T from the ESXi host.