vSAN Component Limit Exhaustion causing "Underlying device has no free space" errors
Article ID: 438233
Updated On:
Products
Issue/Introduction
In VMware vSAN Original Storage Architecture (OSA) environments, operations such as provisioning new Virtual Machines, creating snapshots, or starting replications via VMware Cloud Director Availability (VCDA) may fail with out-of-space errors. These failures occur even when the vSAN datastore displays sufficient physical capacity because the host or cluster has reached its architectural metadata component limit.
Symptoms:
Skyline Health: flags Component Limit Health as Red, while Disk Free Space Health remains Green.
- Operations: Provisioning, snapshots or replications fail with errors indicating no free space.
Logs: ESXi logs (
/var/run/log/vmkwarning.log) report:RPC to DOM op create returned: Underlying device has no free space.textRPC to DOM op create returned: Underlying device has no free space
Cluster And Host Component Utilization section of skyline health reports utilization health in red:
Environment
VMware ESXi 8.0.x
vSAN Original Storage Architecture (OSA)
Cause
The vSAN cluster has exhausted the component limit allowed by the metadata layer. In vSAN Original Storage Architecture (OSA), each host is limited to a hard maximum of 9,000 components. A 4-node cluster, for example, has an aggregate limit of 36,000 components. Snapshots significantly impact this count as each creates additional components for every virtual disk.
Note: This limit does not apply in the same way to vSAN Express Storage Architecture (ESA), which has significantly higher limits.
Resolution
To resolve component exhaustion, follow these steps to reduce component count or increase cluster capacity:
1. Quantify Utilization
- Check host-level limits:
# localcli vsan debug limit get - Calculate total cluster components:
bashlocalcli vsan debug object overview | grep " of " | awk -F'of ' '{sum += $2} END {print "Total Components: " sum}' - Monitor via UI: Navigate to vSAN Cluster > Monitor > Skyline Health > Cluster and host component utilization.
2. Manage Virtual Machine Snapshot Identify VMs with high snapshot counts and consolidate them to reclaim metadata slots:
powershell Get-VM | Get-Snapshot | Group-Object VM | Select-Object Name, Count | Sort-Object Count -DescendingAction: Perform a Delete All snapshots operation on identified VMs.
3. Identify Orphaned Components List inaccessible components not associated with active VMs: # localcli vsan debug object list --health=inaccessible
Action: If inaccessible objects are found, contact Broadcom Support for guided cleanup.
4. Optimize Storage Policies
Stripe Width: Set Number of disk stripes per object to 1.
RAID Configuration: Consider transitioning from RAID-1 to RAID-5/6 (if host count allows) to improve metadata efficiency.
5. Expand the Cluster Add additional ESXi hosts to the cluster. Each new host in an OSA cluster adds 9,000 component slots to the aggregate pool.
Additional Information
Feedback
