vSAN -- Cluster is down after a successful shutdown with the Shutdown Cluster Wizard. Hosts and vCenter will not come back up. vSAN Datastore shows 0B.

Products

• VMware vSAN
• VMware vSAN 7.x
• VMware vSAN 8.x

Issue/Introduction

VMs are showing as "invalid" in the Host Client (incl. vCenter)

 

vSAN Datastore is not showing any VMs

 

Environment

vSAN 7.x

vSAN 8.x

Cause

vSAN Cluster is still in Shutdown State and therefore vSAN Datastore is not available. vSAN not functional.

Resolution

1.) Ensure that none of the vSAN Hosts is in Maintenance Mode

By logging into one of the vSAN Hosts via SSH/Putty und run:

echo "hostname,decomState,decomJobType";for host in $(cmmds-tool find -t HOSTNAME -f json |grep -B2 Healthy|grep uuid|awk -F \" '{print $4}');do hostName=$(cmmds-tool find -t HOSTNAME -f json -u $host|grep content|awk -F \" '{print $6}');decomInfo=$(cmmds-tool find -t NODE_DECOM_STATE -f json -u $host |grep content|awk '{print $3 $5}'|sed 's/,$//');echo "$hostName,$decomInfo";done|sort

Sample output:
hostname,decomState,decomJobType
esxi-1.example.com,0,0
esxi-2.example.com,0,0
esxi-3.example.com,6,0 --> Anything other than "0": Host in vSAN Decom State (= State "6" specifically refers to: Host in Maintenance Mode)

--> For any Host showing with state "6": Open the ESXi Host Client and put the Host out of Maintenance Mode (= Host Client Example: http://esxi-3.example.com/ui )

 

2.) Check the Cluster Power Status

Open SSH/Putty Sessions to all vSAN Hosts and run on each Host the following two commands:

 

esxcfg-advcfg -g /VSAN/DOMPauseAllCCPs
esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates

 

--> Do one or both of the command outputs showing value "1": Proceed with Step 3.) 

Sample output shows both have set "1": 

 

--> If both values are set to "0": Proceed with powering up your vCenter and Step 4.)

 

 

3.)  If one or both of the command outputs are showing value "1"

Execute on the affected Hosts one or both of following commands:

 

If DOMPauseAllCCPs is set to "1":

esxcfg-advcfg -s 0 /VSAN/DOMPauseAllCCPs

 

If IgnoreClusterMemberListUpdates is set to "1":

esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates

 

 

4.) After vCenter VM is up: Log into vCenter vSphere Client

 

 

5.) Check the Shutdown Status from a vCenter Perspective

Navigate to the Cluster --> Configure --> vSAN --> Services --> Hit "Restart"

 

vCenter will now catch up with the Cluster Status by virtually running to through the steps to bring Cluster Status back online in the vSphere Client.

When successfully completed, the vSAN Services page should update into the following:

 

7.) Wait for any Resync Activity to complete

 

8.) Check your Cluster Status by running vSAN Skyline Health

 

9.) If all is looking good, proceed with powering on any needed VMs

 

If assistance is required, please open a Ticket with VMware by Broadcom Support.