Remove workflows will fail if NSX-T services have been enabled on a host cluster

Products

• VMware Cloud Foundation

Issue/Introduction

• This article will helps to unblock them so that users can resume and complete their Remove workflow operations

Symptoms:

• The remove workflows will fail while executing an action, "Enter Maintenance Mode on ESXi Hosts", with error message "host in maintenance mode has VMs running on it".
• Corresponding error statement can be found in log file, /var/log/vmware/vcf/domainmanager/domainmanager.log

Environment

VMware Cloud Foundation 4.0.x

Cause

• NSX-T has a functionality which allows users to deploy some services at cluster level and these services require Transport Node Profile(TNP) to be attached to those clusters.
• In addition to this, these services also deploy some service virtual machines in respective hosts
• In J1 & J2, VCF remove-workflows don't expect TNPs attached to clusters. Also, those service VMs remain deployed in those hosts as long as services are in enabled state in those host clusters which in turn block VCF remove-workfow operations.
   

Resolution

• This issue has been fixed in K1 (VCF 4.1)

Workaround:

• Remove Host. :

Using VC UI, delete the service VM present on the host and retry the failed "remove host" workflow.

• Remove Cluster :

1. Using NSX-T UI, delete the GI service deployment of that specific cluster which is being deleted.
2. Also, if it's a VUM cluster, make sure that TNP is detached from that cluster, using NSX-T UI. In case of vLCM cluster, let it remain attached.
3. Restart the failed workflow.

• Remove Domain : 

1. Using NSX-T UI, delete the GI service deployments of all the clusters under the domain which is being deleted.
2. Also, if those are VUM clusters, make sure that TNP is detached from all those clusters, using NSX-T UI. In case of vLCM clusters, let them remain attached.
3. Restart the failed workflow.