vSAN -- After Power Outage/Maintenance event - All VMs showing as "Invalid". vCenter is down. Storage is empty.

Products

• VMware vSAN
• VMware vSAN 7.x
• VMware vSAN 8.x

Issue/Introduction

Despite everything is back online after a Power Outage/Maintenance:

VMs are showing as "invalid" in the Host Client

 

vSAN Datastore is not showing any VMs

 

Note: vCenter might be located on the affected vSAN Datastore.

Characteristics of the issue:

• The vSAN production VMs are down after a maintenance activity, which might have included a vSAN cluster shutdown and restart.
• VMs on the vSAN cluster are displayed as "invalid" in ESXi host client.
• Inaccessible objects are observed in esxcli vsan debug object health summary get command output.
• One of the vSAN nodes has "Maintenance Mode State: On" in esxcli vsan cluster get command output.
• vSAN might show multiple files missing

 

 

 

Environment

VMware vSAN (All Versions)

Cause

One or more vSAN Hosts are in vSAN Decom State See KB vSAN Host Maintenance Mode is not in sync with vSAN Node Decommission State

 

In order to check, open SSH/Putty Session to all Hosts and run:  

 

esxcli vsan cluster get

You can also run the below script from one of the hosts in the cluster to see which hosts in the cluster are in vSAN Decom state.

echo "hostname,decomState,decomJobType";for host in $(cmmds-tool find -t HOSTNAME -f json |grep -B2 Healthy|grep uuid|awk -F \" '{print $4}');do hostName=$(cmmds-tool find -t HOSTNAME -f json -u $host|grep content|awk -F \" '{print $6}');decomInfo=$(cmmds-tool find -t NODE_DECOM_STATE -f json -u $host |grep content|awk '{print $3 $5}'|sed 's/,$//');echo "$hostName,$decomInfo";done|sort

Sample output:
hostname,decomState,decomJobType
esxi-1.example.com,0,0
esxi-2.example.com,0,0
esxi-3.example.com,0,0
Anything other than 0 means there is a host in vSAN Decom State.
hostname,decomState,decomJobType
esxi-1.example.com,0,0
esxi-2.example.com,0,0
esxi-3.example.com,6,0 <---

Resolution

1. Place any of the vSAN Hosts found in vSAN Decom into Maintenance Mode with "No Action" or "No Data Migration" (= depending on Version).
2. Once the task completes, take the host out of Maintenance Mode.
3. Repeat this process for any other hosts in the cluster that are still in vSAN Decom state
4. Re-evaluate whether the reported issues described via section Issue/Introduction still exist

If assistance is required, please open a Ticket with VMware by Broadcom Support.

For more information, please refer to: A node is in vSAN Maintenance Mode, even though the UI shows the ESXi host is not.