vSAN Cluster Full in Horizon View Environment with Deduplication and Compression

Products

• VMware vSAN

Issue/Introduction

• You receive continuous pop-up windows in the vCenter GUI or a local host web client with the message, "There is no more space for virtual disk <vmdk_name.vmdk>. You might be able to continue th esession by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate the session."
  
  
• You experience a sudden drastic increase in capacity utilization of your vSAN cluster
• Your vSAN cluster is over 95% full
  
• There is an ongoing resync
• VMs are unresponsive

 

 

Environment

vSAN OSA with deduplication enabled

Cause

The following operations occur resulting in the datastore showing space has become available before the space has been physically freed on disks.

• An operation such as a recompose is performed resulting in a large amount of vSAN objects being deleted
• Deduplication pending deletes are draining slowly resulting in space not yet being physically available on disks
• A large deployment of instant clone cp-replicas begins
• Large vswap files created as part of the Horizon provisioning process
• Backup jobs activated at the time of the provisioning

Once the disks have become unavailable for use due to capacity being filled VMs will stop or hang and the prompt will appear informing that there is not space available for virtual disk.

VMs will not be able to function until space is freed.

 

Resolution

Along with following the KB of Resolving Storage Capacity Issues in a vSAN Datastore ensure that you:

• Run esxcfg-advcfg -s 99 /VSAN/ClomRebalanceThreshold to stop rebalance activity
• Stop all backup operations to prevent additional related data from being placed
• Convert thick provisioned objects to thin to release reserved space
• Clean up unused objects including unassociated objects and replicas
• Convert VMs from RAID 1 to RAID 0 to release half of the used space (do not convert RAID 5 or 6 objects as this will result in additional space utilization while data is rebuilt into the new format)
• Allow deletes to finish freeing space (monitor capacity utilization)
• Ran esxcfg-advcfg -s 80 /VSAN/ClomRebalanceThreshold to rerestart rebalancing and allow disks still near full to properly balance data to less full disks
• Verify the cluster is stable
• Change RAID 0 objects back to RAID 1 while monitoring capacity
  Note: It's recommended not to change the policy for all objects at once to avoid any potential space/performance issues while relaying out the objects. Do the policy change in small batches