Deleting a cluster in a NSX-T Workload Domain fails at "Transport Node deletion operation failed"

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• When trying to delete a cluster from a NSX-T Workload Domain, the workflow fails with "Transport Node deletion operation failed". 
• In the /var/log/vmware/vcf/domainmanager/domainmanager.log, similar exceptions messages are seen:

2019-09-16T12:56:13.333+0000 ERROR [6004dec066944594,860b] [c.v.v.c.f.p.nsxt.helpers.NsxtUtils,dm-exec-5] Error while getting transport node deletion task result for Host esxi-5.vrack.vsphere.local
java.util.concurrent.TimeoutException: null

• In the NSX-T Manager UI, similar error messages to the below are seen:

Failed to uninstall the software on host. Unable to remove vib on host. java.rmi.RemoteException: [LiveInstallationError] Error in running [‘/etc/init.d/nsx-datapath’, ‘stop’, ‘remove’]: Return code: 1 Output: stop remove begin Exception: Traceback (most recent call last): File “/etc/init.d/nsx-datapath”, line 1404, in <module> raise Exception(str(numOpaqueSwitches) + ” opaque switch(es) present”) Exception: 1 opaque switch(es) present It is not safe to continue. Please reboot the host immediately to discard the unfinished update. Please refer to the log file for more details.

Note: This log excerpt is an example. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 3.9.x

Resolution

This is known issue in NSX-T 2.5.0. The issue is resolved in NSX-T 2.5.1, which will be part of a future VMware Cloud Foundation version. 

Workaround:

To workaround this issue, please see the steps below:

1. Login to the NSX-T Manager UI and navigate to System -> Nodes -> Host Transport Node
2. Select the cluster that is being deleted from NSX-T Workload Domain.
3. Find and select the host that is in error state, then click the REMOVE NSX button
4. In the pop up window, un-select the Uninstall NSX components checkbox option and select the Force Delete checkbox option.
5. Navigate back to the SDDC manager UI and retry failed workflow.