Uninstalling NSX-T from ESXi host failing with error "Failed to remove all host switches or logical switches"

Products

• VMware NSX

Issue/Introduction

• While attempting to install NSX-T on host, installation fails at 18% with the following errors:
  

  "failed to install software on a host <hostname>:java.rmi.RemoteExcepion:
  
  [Live installation error] Error in running ['/rtc/init.d/nsx-opsagent'. 'stop'. 'upgrade']: Return code : 1 Output OK to upgrade nsx-opsagent stop nsx-opsagent stop watchdog
  
  Terminating watchdog process with process PID 2105211 sh: you need to specify whom to kill nsx-ops-agent service is stopping cp: can't stat"

• The resolve option does not help, if you attempt to run 'del nsx' in the nsxcli of the ESXi host, results in the below errors:

  Exception when deleting nsx from host: ' error code: 4 stdout: delete_nsx_instance_from_host.sh: INFO: NSX reset script called with argument fabric_node on nsx-esx delete_nsx_instance_from_host.sh: INFO: Run transport_node reset on ESX node % Failed to remove all host switches or logical switches delete_nsx_instance_from_host.sh: ERROR: Failed to reset nsxa app of nsx-opsagent. Please check ospagent logs for more details. , stderr: <date-time> ERROR: Failed to reset nsxa app of nsx-opsagent. Please check ospagent logs for more details."

• The /var/run/log/esxupdate.log on the ESXi host shows vdl2 unload failed errors:

  cpu48:4580298)Mod: 5098: Unloading module <vmk-module-uuid> ...
  
  cpu48:4580298)vdl2: VDL2Cleanup:756: [nsx@6876 comp="nsx-esx" subcomp="<vmk-module-uuid>"]Starting cleanup
  
  cpu48:4580298)ALERT: Mod: 5251: Failed to unload module <vmk-module-uuid>, since its consumed resource count is 1. Waiting...
  
  cpu48:4580298)ALERT: Mod: 5280: Failed to unload module <vmk-module-uuid>, since its consumed resource count is

• Host properties are set to true on the DVS, which can be seen by running net-dvs -l.
  

  com.vmware.nsx.kcp.enable
  
  com.vmware.nsx.spf.enabled
  
  com.vmware.nsx.vdl2.enabled
  
  com.vmware.net.portset.fc.enabled
  
  com.vmware.net.portset.fc.mcast.enabled

• There is at least one host switch with the following property set to down

  net-dvs -l | grep -E "com.vmware.common.alias|com.vmware.common.opaqueDvs.status.component.vswitch"
  Example output:
  com.vmware.common.alias = <DVS name> ,        propType = CONFIG
  com.vmware.common.opaqueDvs.status.component.vswitch=down

• /var/run/log/nsxavim.log may also show entries similar to the below on the host:
  

  INFO HowSwapCvds start: cvds [<DVS UUID>], retry-times 60
  
  INFO Nsx host properites remain in dvs [<DVS UUID>]: ['com.vmware.net.portset.fc.enabled','com.vmware.net.fc.mcast.enabled'] 
  
  WARNING NSX properties remain in VDS [<DVS UUID>] are not removed yet.

Cause

The issue is caused when one or more of the modules com.vmware.nsx.spf, com.vmware.nsx.vdl2, com.vmware.nsx.kcp are loaded and enabled and preventing the NSX VIB's from being removed.

Resolution

This is a known issue affecting VMware NSX

• To resolve the issue, remove the stale entries before re-preparing the host for NSX by following NSX VIBs are not removed from an ESXi host.
• Once the stale entries are removed, you can proceed with re-installing NSX on the affected ESXi host through NSX UI.