vSAN -- VMs on vSAN show as "Invalid" and/or inaccessible after power outage or completion of maintenance task

Products

• VMware vSAN
• VMware vSAN 7.x
• VMware vSAN 8.x

Issue/Introduction

You observe one or more of the following Symptoms after ESXi Hosts came back online after a power outage or a maintenance task:

• The vSAN production VMs are down after a maintenance activity, which might have included a vSAN cluster shutdown and restart.
• VMs on the vSAN cluster are displayed as  "invalid" or "unknown" in the ESXi host client.
• Inaccessible objects are observed in output of the command:  esxcli vsan debug object health summary get
• vSAN shows multiple files missing
• Multiple Virtual Machine shows as "Inaccessible" in vCenter

 

• The vCenter VM located on the affected vSAN Datastore might not be accessible

• VMs still show as unknown or invalid in client view:

 

• vSAN Datastore view may not show any VMs:

 

 

 

Environment

VMware vSAN (All Versions)

Cause

The vSAN Healthcheck reveals that one or more vSAN Hosts are in vSAN Decommission State as described in KB 318411

 

 

 

This can be verified via SSH/Putty Session as well: 

 

esxcli vsan cluster get

You can also run the below script from one of the hosts in the cluster to see which hosts in the cluster are in vSAN Decom state.

echo "hostname,decomState,decomJobType";for host in $(cmmds-tool find -t HOSTNAME -f json |grep -B2 Healthy|grep uuid|awk -F \" '{print $4}');do hostName=$(cmmds-tool find -t HOSTNAME -f json -u $host|grep content|awk -F \" '{print $6}');decomInfo=$(cmmds-tool find -t NODE_DECOM_STATE -f json -u $host |grep content|awk '{print $3 $5}'|sed 's/,$//');echo "$hostName,$decomInfo";done|sort

Sample output:
hostname,decomState,decomJobType
esxi-1.example.com,0,0
esxi-2.example.com,0,0
esxi-3.example.com,0,0
Anything other than 0 means there is a host in vSAN Decom State.
hostname,decomState,decomJobType
esxi-1.example.com,0,0
esxi-2.example.com,0,0
esxi-3.example.com,6,0 <---

Resolution

1. Place any of the vSAN Hosts found in vSAN Decom into Maintenance Mode with "No Action" or "No Data Migration" (= depending on Version).
2. Once the task completes, take the host out of Maintenance Mode.
3. Repeat this process for any other hosts in the cluster that are still in vSAN Decom state
4. Re-evaluate whether the reported issues described via section Issue/Introduction still exist

If assistance is required, please open a Ticket with VMware by Broadcom Support.

For more information, please refer to: A node is in vSAN Maintenance Mode, even though the UI shows the ESXi host is not.