After vSAN Cluster Shutdown & Restart, VMs/objects are inaccessible

Products

VMware vSAN

Issue/Introduction

The vSAN Cluster was shutdown via the Shutdown Wizard in the vSphere Web Client for planned cluster maintenance.

After powering on the hosts of the vSAN Cluster all VMs show as inaccessible in the vCenter Web Client (if available) and as Invalid in the Host UI

vCenter Web Client:

Host UI:

vSAN Datastore is not showing any VMs

After clicking the Restart Button in vCenter Web Client (if available) the task doesn't complete successfully.

Hosts show as disconnected in the vCenter Web Client.

Environment

VMware vSAN (All versions).

Cause

As part of the vSAN Cluster Shutdown Wizard the advanced parameters "/VSAN/DOMPauseAllCCPs" and "/VSAN/IgnoreClusterMemberListUpdates" are set to "1" on all hosts in the vSAN cluster in order to preserve data in case of any potential hardware failures during cluster restart. Upon starting the vSAN cluster, the cluster Restart task is expected to change these values back to the default setting of "0".

If something prevents the script from being able to run on its own, such as a host failing to power back on or the hosts disconnected from vCenter, then the values will not be set back to the default value of 0 and this will need to be set back manually to bring production back online.

The advanced setting "/VSAN/DOMPauseAllCCPs" set to 1 pauses all vSAN components operations on the host, effectively making objects owned by that host inaccessible.

In other words, the vSAN objects are not available to the compute resources, resulting in inaccessible VMs in the inventory.

Resolution

Modify the advanced parameter DOMPauseAllCCPs and IgnoreClusterMemberListUpdates on all the hosts in the cluster to the default setting of 0.

Run command "esxcfg-advcfg -g /VSAN/DOMPauseAllCCPs" on all hosts to get the current value:

[root@esxi#:~] esxcfg-advcfg -g /VSAN/DOMPauseAllCCPs
Value of DOMPauseAllCCPs is 1

If the value is 1, this means /VSAN/DOMPauseAllCCPs is enabled and the objects owned by this host will be unavailable until the setting is reverted to the default value of 0 by running the following command on the affected hosts.:

[root@esxi#:~] esxcfg-advcfg -s 0 /VSAN/DOMPauseAllCCPs

Run command "esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdate" on all hosts to get the current value:

[root@esxi#:~] esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates
Value of IgnoreClusterMemberListUpdateis 1

If the value is 1, this means /VSAN/IgnoreClusterMemberListUpdate is enabled and vCenter will not be able to make any changes to cluster membership until the setting is reverted to the default value of 0 by running the following command on the affected hosts.:

[root@esxi#:~] esxcfg-advcfg -s 0 /VSAN/ IgnoreClusterMemberListUpdates