Unable to snapshot VMs when less than the minimum required hosts of a specific storage policy are online

Products

• VMware vSAN

Issue/Introduction

When trying to create a snapshot of a VM residing on a vSAN datastore, it fails with the error:

There are currently 1 usable fault domains. The operation requires 2 more usable fault domains.
An error occurred while taking a snapshot: Out of resources.

This is the case when there are fewer hosts available than required to satisfy a specific storage policy (e.g., 2 hosts on the default FTT-1 instead of 3). They may either have failed or have been put into maintenance mode.

 

Environment

VMware vSAN (All Versions)

Resolution

vSAN requires a minimum number of hosts to satisfy its storage policy, depending on the specific policy. Examples are:

3 hosts for default FTT=1 RAID 1
4 hosts for FTT=2 RAID 1
4 hosts for FTT=1 RAID5 Erasure Coding
...

Please refer to the following documentation regarding Fault Domains for further information on vSAN cluster sizing. For safety reasons, in its default configuration vSAN will not allow the provisioning of new objects if there are not enough resources to satisfy the applied storage policy.

To resolve this, run the below command and make sure there are no hosts in Decom State.

echo "hostname,decomState,decomJobType";for host in $(cmmds-tool find -t HOSTNAME -f json |grep -B2 Healthy|grep uuid|awk -F \" '{print $4}');do hostName=$(cmmds-tool find -t HOSTNAME -f json -u $host|grep content|awk -F \" '{print $6}');decomInfo=$(cmmds-tool find -t NODE_DECOM_STATE -f json -u $host |grep content|awk '{print $3 $5}'|sed 's/,$//');echo "$hostName,$decomInfo";done|sort

Sample output:
hostname,decomState,decomJobType
esxi-1.example.com,0,0
esxi-2.example.com,0,0
esxi-3.example.com,0,0
Anything other than 0 means there is a host in decommissioned mode
hostname,decomState,decomJobType
esxi-1.example.com,0,0
esxi-2.example.com,0,0
esxi-3.example.com,0,6 <---

If you find a host in Decom State, validate no VMs are powered-on on it and place the host into maintenance mode ('No Action' option), and then remove the host from maintenance mode using the host UI or vCenter UI to clear this state.

Workaround:

If a host has failed or can't be taken out of maintenance mode, as a workaround, "Force provisioning" can be temporarily enabled by setting the option to "Yes" in the settings of the specific storage policy.

Please keep in mind that in this case, objects can be created even if they do not satisfy the requirements of a specific storage policy, so it is recommended to change this back after the host is back online again.

Additional Information

See also:Sizing for Capacity Maintenance and Availability