vSAN Cluster shutdown wizard doesn't work when using vCenter HA

Products

• VMware vSAN

Issue/Introduction

To advise if vCenter HA is in use and if you need to shut down your vSAN cluster to shut down the cluster with workarounds listed here.

Symptoms:

• vCenter HA is in use
• vSAN running 7.0U3c or higher but below 8.0 the Shutdown pre-check page has a red item -- "Some customer VMs are not powered off." The page will also list the virtual machines for vCenter Passive and Witness nodes for this check.
• For vSAN 8.0 and higher, the cluster shutdown process fails after vCenter VM is powered off. And in /var/run/log/vsanmgmt.log on the orchestration host where vCenter Active Node is running, you will see the error "PerformOrchestrationClusterPowerAction error: currentPowerStatus: hostsInfraVMsPoweredoff".

Environment

VMware vSAN 8.0.x
VMware vSAN 7.0.x

Cause

vSAN cluster shutdown wizard doesn't handle shutting down the service VMs well when vCenter HA is in use. This is due to vCenter being configured as a three-node cluster that contains Active, Passive, and Witness nodes for failover capabilities.

If the vCenter Passive node and vCenter Witness node are not powered off before shutting down the cluster and instead allow the shutdown process to power them off, the shutdown workflow will fail due to the timeout caused by powering off the vCenter Passive node.

Resolution

VMware engineering is aware of the issue and is working on a fix for a future release.

Workaround:
If the vSAN version is 7.0U3c or higher but below version 8.0 then follow Manually Shut Down and Restart the vSAN Cluster after shutting down the vCenter HA nodes as per Reboot All vCenter HA Nodes.

For vSAN clusters running version 8.0 or higher follow the below steps.
1. Power off the vCenter Passive node VM
2. Use the cluster shutdown wizard to shut down the cluster and allow the shutdown workflow to power off the vCenter Active Node and Witness node

To restart the cluster after shutting it down follow these steps:
1. Power on the hosts in the cluster
2. Wait for vCenter Server to restart
3. click "Restart cluster" in the vSphere Client
4. Power on the vCenter Passive node VM

Additional Information

Shut Down the vSAN Cluster Using the Shutdown Cluster Wizard

For more information regarding vCenter HA see the below documents:
How vSphere HA Works
vCenter High Availability
FAQ: vCenter High Availability (2148003)

Impact/Risks:
vSAN cluster shutdown wizard fails to shut down the vSAN cluster.