Removing hosts from a stretched cluster in VMware Cloud Foundation fails on the Get Data from Inventory task

Products

• VMware Cloud Foundation

Issue/Introduction

Symptoms:

• Attempting to remove hosts from a stretched cluster in VMware Cloud Foundation fails with error "NSX Manager is in a bad state"
• The hosts being removed are no longer accessible in the vSphere Web Client.
• The Host Preparation status of the hosts being removed is not Green on the Networking and Security page in the vSphere Web Client.
• You see messages similar to the following in the /var/log/vmware/vcf/domainmanager/domainmanager.log file on the SDDC  Manager VM:

2018-10-28 17:42:09.366 [vcf_dm,71f5e86cc26f39f7,018eb553f2a45aac] [-thread-18] DEBUG [ c.v.v.h.action.RemoveHostEntityStatusActionUtil] Validate NSX manager status
2018-10-28 17:42:09.367 [vcf_dm,71f5e86cc26f39f7,018eb553f2a45aac] [-thread-18] ERROR [c.v.e.sddc.orchestrator.model.error.ErrorFactory] [POA21U] REMOVE_HOSTS_NOT_ACTIVE_NSX_MANAGER_ERROR Not attempting to remove hosts esx-1.vrack.vcf.local,esx-10.vrack.vcf.local,esx-3.vrack.vcf.local,esx-4.vrack.vcf.local,esx-2.vrack.vcf.local from cluster SDDC-Cluster1 when the NSX Manager is in a bad state. Retry after confirming NSX Manager is not reporting any errors.
com.vmware.evo.sddc.orchestrator.exceptions.OrchTaskException: Not attempting to remove hosts esx-1.vrack.vcf.local,esx-10.vrack.vcf.local,esx-3.vrack.vcf.local,esx-4.vrack.vcf.local,esx-2.vrack.vcf.local from cluster SDDC-Cluster1 when the NSX Manager is in a bad state. Retry after confirming NSX Manager is not reporting any errors.
        at com.vmware.vcf.hostmanager.action.RemoveHostEntityStatusActionUtil.validateEntityStatuses(RemoveHostEntityStatusActionUtil.java:58)
        at com.vmware.vcf.hostmanager.action.RemoveHostGetInventoryAction.execute(RemoveHostGetInventoryAction.java:162)
        at com.vmware.vcf.hostmanager.action.RemoveHostGetInventoryAction.execute(RemoveHostGetInventoryAction.java:76)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.lambda$static$0(FsmActionState.java:14)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionState.invoke(FsmActionState.java:62)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:167)
        at com.vmware.evo.sddc.orchestrator.platform.action.FsmActionPlugin.invoke(FsmActionPlugin.java:154)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.invokeMethod(ProcessingTaskSubscriber.java:340)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.processTask(ProcessingTaskSubscriber.java:440)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:98)
        at com.vmware.evo.sddc.orchestrator.core.ProcessingTaskSubscriber.accept(ProcessingTaskSubscriber.java:55)
        at reactor.bus.EventBus$3.accept(EventBus.java:317)
        at reactor.bus.EventBus$3.accept(EventBus.java:310)
        at reactor.bus.routing.ConsumerFilteringRouter.route(ConsumerFilteringRouter.java:72)
        at reactor.bus.EventBus.accept(EventBus.java:591)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware Cloud Foundation 3.5.x

Resolution

This is a known issue affecting VMware Cloud Foundation. There is currently no resolution.

Workaround:
Note: Do not attempt to use the Resolve functionality on the Networking and Security page in the vSphere Web Client for any of the affected hosts.

1. Disconnect the affected ESXi hosts in the vSphere Web Client by right-clicking each and choosing Disconnect under the Connection menu. Click the Yes button on the confirmation dialogue. 
2. Right-click each affected ESXi host and choose Remove From Inventory. Click the Yes button on the confirmation dialogue.
3. Verify that the cluster status on the Networking and Security page in the vSphere Web Client is now Green.
4. Attempt the host removal operation again in the SDDC Manager UI.

Additional Information

To be alerted when this article is updated, click the Subscribe to Article link in the Actions box.