VxRail Host Removal Operation from vCenter fails for Multi-DVS cluster

Products

• VMware vCenter Server

Issue/Introduction

Symptoms:

• vCenter UI shows remove host pre-check failures.
• VxRail web.log shows following error messages;

2020-08-18T06:51:21.868+0000 INFO [Thread-667] com.emc.mystic.manager.web.service.nodes.prechecker.VsanHealthyChecker VsanHealthyChecker.check:18 - VsanHealthyChecker Cluster Status = CRITICAL
2020-08-18T06:51:21.872+0000 INFO [Thread-667] com.emc.mystic.manager.commons.db.util.OperationStatusUtil OperationStatusUtil.updateOperationStatusInfo:72 - Updating operation status info in the DB: c68b08a6-f225-452c-ae9f-3494d26a1ae1
2020-08-18T06:51:21.874+0000 INFO [Thread-667] com.emc.mystic.manager.web.service.nodes.action.PrecheckAction PrecheckAction.perform:48 - Failed to do PrecheckAction...
2020-08-18T06:51:21.874+0000 INFO [Thread-667] com.emc.mystic.manager.web.service.nodes.action.PrecheckAction PrecheckAction.getErrorMessage:79 - Node removal precheck error bundleName: NodeOperation, key: node.removal.vsanhealty.check, params: [], locale: en_US.
2020-08-18T06:51:21.874+0000 ERROR [Thread-667] com.emc.mystic.manager.web.service.nodes.action.PrecheckAction PrecheckAction.perform:51 - Failed to do PrecheckAction. Name: VsanHealthyChecker, Error: Failed to check vSAN health.
2020-08-18T06:51:21.874+0000 ERROR [Thread-667] com.emc.mystic.manager.web.service.nodes.NodeRemovalServiceImpl NodeRemovalServiceImpl.perform:152 - Failed to Remove node
com.emc.mystic.manager.web.webutil.RestException
        at com.emc.mystic.manager.web.webutil.RestException.NodeRemovalPrecheckFailedException(RestException.java:449) ~[classes/:?]
        at com.emc.mystic.manager.web.service.nodes.action.PrecheckAction.perform(PrecheckAction.java:54) ~[classes/:?]
        at com.emc.mystic.manager.web.service.nodes.action.AbstractNodeAction.execute(AbstractNodeAction.java:39) ~[classes/:?]
        at com.emc.mystic.manager.web.service.nodes.NodeRemovalServiceImpl.perform(NodeRemovalServiceImpl.java:143) ~[classes/:?]
        at com.emc.mystic.manager.web.controller.ClusterControllerImpl.performNodeRemoval(ClusterControllerImpl.java:155) ~[classes/:?]
        at com.emc.mystic.manager.web.controller.ClusterControllerImpl.lambda$removeHost$1(ClusterControllerImpl.java:118) ~[classes/:?]
        at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640) [?:1.8.0_241]

Resolution

The Workaround is the removal of hosts from overlay DVS.

1. Go to Networks Tab in vCenter UI.
2. Right-click on Secondary (Overlay) DVS.
3. Select Add/Manage Hosts Option.
4. From the dialog box select "Remove Host" radio button and click on Next.
5. Click on Attached Hosts and select the host to be removed from the host list.
6. Click on OK.
7. Click on Next.
8. Click on Finish.
9. Retry host removes operation.

Host remove operation starts and can be monitored from the vCenter tasks window.