vSAN Cluster Operations Fail: Provisioning, Snapshots, and Cloning Blocked Due to Host Rebuild Capacity Exhaustion

Products

• VMware vSAN

Issue/Introduction

Cloning, snapshot creation, or VM migration tasks fail on the vSAN datastore even when space appears available. This occurs when the cluster utilized capacity reaches a protection threshold that blocks further provisioning to preserve data resiliency.

Symptoms:

• vSAN Skyline Health reports 'Storage space' or 'What if the most consumed host fails' alarms.
• Task fails with the error: 

There are currently 0 bytes usable space. The operation requires ####### more bytes of usable space. An error occurred while taking a snapshot: Out of resources.

• The failure log messages are captured under var/run/log/hostd.log:

YYYY-MM-DDThh:mm:ss.ms Db(167) Hostd[#####]: [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/vsan:####.vmx] Create Snapshot message: There are currently 0 bytes usable space.The operation requires ####### more bytes of usable space.
YYYY-MM-DDThh:mm:ss.ms Db(167) Hostd[#####]: --> An error occurred while taking a snapshot: Out of resources.

• vSAN File Services Impact: The capacity exhaustion issue can also block the deployment or remediation of vSAN File Services. The vSAN File Services VM (FSVM) may fail to deploy or remediate as the underlying storage policy cannot satisfy requirements due to the reserved space.

Environment

VMware vSAN 8.0 OSA

VMware ESXi 8.0.x

Cause

• This issue occurs because the vSAN cluster has reached or exceeded its Host Rebuild Capacity Reservation Threshold.

• vSAN reserves a portion of the total datastore capacity specifically to guarantee that if any single host fails, the system has sufficient space to fully rebuild all impacted components onto the remaining hosts in the cluster, maintaining the required Primary Level of Failures to Tolerate (PFTT).

Validation:

• var/run/log/clomd.log contains the error message:

YYYY-MM-DDThh:mm:ss.ms No(29) clomd[2099188]: [Originator@6876 opID=########] CLOMExceedsCapacityReservationThreshold: usedCapacity(#########) exceeds hostRebuildThreshold(#########)
YYYY-MM-DDThh:mm:ss.ms No(29) clomd[2099188]: [Originator@6876 opID=########] CLOM_Diagnose: There are currently 0 bytes usable space.The operation requires ########### more bytes of usable space.
YYYY-MM-DDThh:mm:ss.ms No(29) clomd[2099188]: [Originator@6876 opID=########] CLOMProcessWorkItem: Failed to generate configuration: Underlying device has no free space

Resolution

1. Reclaim Immediate Capacity:

   • Identify and delete large, unused, or decommissioned Virtual Machines to reach 80% utilization.
   • Consolidate or delete old snapshots.
   • Remediate or remove inaccessible objects.

2. Permanent Capacity Expansion:

   • Add new disks (Scale-up) to existing disk groups across all hosts.
   • Add new hosts (Scale-out) to the vSAN cluster.

3. Emergency Workaround (Temporary):

   • Caution: Disabling Host Rebuild Reserve allows provisioning and management operations to succeed by freeing reserved space. This leaves the cluster vulnerable. If a host fails while the reserve is disabled, the remaining hosts lack sufficient space to rebuild objects, leading to data non-compliance. Perform this step for emergency recovery only and reverse it once capacity is available.
   • Navigate to the vCenter Server > vSAN Cluster > Configure tab.
   • Under vSAN, select Services > Advanced Options.
   • Disable the Host rebuild reserve setting.
   • Resume the failed cloning or snapshot operation.

Additional Information

vSAN Health Service - Capacity utilization - Storage space