Virtual machine stops with "There is no more space for virtual disk" on NFS or VMFS Datastores

Products

• VMware vSphere ESXi

Issue/Introduction

• A virtual machine stops responding when a datastore exhausts its physical capacity. This often occurs during thin-provisioned disk growth or snapshot consolidation, which requires additional "working space" overhead to merge delta blocks. In some cases, a storage discrepancy may be observed where the Guest OS reports free space while the vSphere datastore indicates it is full.
• Powering on VM gives below error:

  There is no more space for virtual disk <VM.vmdk>. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session.

  • 
• The virtual machine stops responding. If the VM is running on a snapshot, the consolidation fails.
• Storage vMotion fails with a pending question: 

  msg.hbacommon.outofspace

•  VM exhibits 99.999% CPU Contention or high %WAIT in esxtop, despite low CPU demand within the guest OS.
• Linux Guest OS gives below Warnings:

  kernel:[.......] Buffer I/O error on device dm-0, logical block 500054 kernel:[.......] blk_update_request: I/O error, dev sda, sector 1213223233
  

• /vmfs/volumes/<datastore_name>/<vm_name>/vmware.log on ESXi gives below error:

  YYYY-MM-DDTHH:MM:SS.708+0000In(05) vmx - [msg.hbacommon.outofspace] There is no more space for virtual disk '/vmfs/volumes/DATASTORE_LOCATION/VM_NAME/VM_DISK.vmdk'.

• /var/run/log/vmkernel.log gives below error:

  cpu24:887###3)Fil6: 4094: 'Datastore_Name': Fil6 file IO () : No space left on device Error message on the UI : There is no more space for Virtual disk.
  

  
  
  

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

The VMFS datastore has exhausted all Small File Blocks (SFBs) and Large File Blocks (LFBs), halting new block allocations for VM writes and swap files. Because ESXi requires temporary free space (overhead and metadata) to merge delta disks into a base disk, snapshot consolidation fails entirely when this "working space" is unavailable.

Resolution

Follow the below steps to resolve the issue:

1. Clear unwanted data, such as old log bundles and ISO images, to immediately free up space. Delete an ISO Image from the vSphere Lifecycle Manager Depot
2. Expand the LUN capacity from the backend storage array and increase the VMFS datastore size.Increase Capacity of vSphere VMFS Datastore
3. Consolidate or delete existing virtual machine snapshots. Note that a 20% free space buffer is recommended for stable thin-provisioning and consolidation operations. Consolidate Snapshots
4. Perform a Storage vMotion of the virtual machine to a datastore with more free space.How to Migrate Your Virtual Machine with Storage vMotion
5. Clone the virtual machine disk (VMDK) to another datastore using the vmkfstools utility. Cloning and converting virtual machine disks with vmkfstools
6. Add a temporary memory reservation to the virtual machine if the swap file is too large. Allocate Memory Resources to a Virtual Machine
7. Perform a cluster level storage rescan on all the clusters the datastore is connected to after increasing the space from the backend. Refer following kb: Performing a rescan of the storage on an ESXi host

 

Additional Information