vSAN Cluster Operations Fail: Provisioning, Snapshots, and Cloning Blocked Due to Host Rebuild Capacity Exhaustion

Products

• VMware vSAN

Issue/Introduction

Symptoms:

• You may not be able to create New Virtual Machines (VMs) on the vSAN datastore.
• Snapshots creation may Fail for existing VMs residing on the vSAN datastore.
• Data Movement Tasks may fail, including VM migration (Storage vMotion) and VM cloning.
• vSAN Health Proactive Tests may report failure with the error: "Failed to create Object. There are currently 0 bytes usable space. the operation failed."

Environment

VMware vSAN 8.x

Cause

This issue occurs because the vSAN cluster has reached or exceeded its Host Rebuild Capacity Reservation Threshold.

vSAN reserves a portion of the total datastore capacity specifically to guarantee that if any single host fails, the system has sufficient space to fully rebuild all impacted components onto the remaining hosts in the cluster, maintaining the required Primary Level of Failures to Tolerate (PFTT).

Validation:

You may see the following log entries from /var/run/log/clomd.log:

CLOMExceedsCapacityReservationThreshold: usedCapacity(42991887167162) exceeds hostRebuildThreshold(37942698762240)

Resolution

To restore operational capabilities while a permanent storage capacity increase is being implemented, follow these steps:

Step 1: Temporary Workaround (Disable Host Rebuild Reserve)

Caution: Disabling the Host Rebuild Reserve temporarily allows provisioning and management operations to succeed by freeing up the reserved space. However, this leaves the cluster vulnerable. If a host were to fail while the reserve is disabled, the remaining hosts may not have enough space to fully rebuild all objects, potentially leading to data non-compliance. This step must only be performed for immediate recovery and must be reversed immediately after Step 2 is completed.

1. Access the vCenter Server.

2. Navigate to the vSAN cluster Configure tab.

3. Under vSAN, select Services.

4. Locate the Advanced Options section (or the equivalent setting for Capacity Management).

5. Disable the Host rebuild reserve setting.

6. Verify that all vSAN operations (VM creation, snapshots) are now succeeding.
   
   

Step 2: Permanent Fix (Increase Cluster Capacity)

The underlying cause is insufficient capacity. The permanent resolution is to expand the vSAN datastore storage pool:

1. Add new disks (Capacity Tier devices) to existing disk groups across all hosts in the vSAN cluster. OR

2. Add new hosts with appropriate disk groups to the vSAN cluster.
   
   

Step 3: Re-enable Host Rebuild Reserve

Once the new capacity has been added and vSAN has finished balancing and utilizing the new space:

1. Navigate back to the vSAN cluster Configure tab > Services > Advanced Options.

2. Re-enable the Host rebuild reserve setting.

3. Monitor the vSAN cluster capacity utilization to ensure it remains below the Host Rebuild Capacity Reservation Threshold.

Additional Information

When the amount of used capacity on the vSAN datastore exceeds this pre-calculated host rebuild threshold, vSAN internally considers the datastore to have "zero bytes usable space" for new provisioning or data modification operations (like snapshots), which could consume additional protected space.