Virtual machine stops with "There is no more space for virtual disk" on NFS or VMFS Datastores
search cancel

Virtual machine stops with "There is no more space for virtual disk" on NFS or VMFS Datastores

book

Article ID: 318352

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

A virtual machine stops responding when a datastore exhausts its physical capacity. This often occurs during thin-provisioned disk growth or snapshot consolidation, which requires additional "working space" overhead to merge delta blocks. In some cases, a storage discrepancy may be observed where the Guest OS reports free space while the vSphere datastore indicates it is full.

Symptoms

  • The virtual machine stops responding.
  • If the VM is running on a snapshot, the consolidation fails.
  • Storage Discrepancy: The Linux guest OS shows free capacity (e.g., 1TB free), but vCenter and the storage array alert that the capacity is exhausted.
  • In the virtual machine summary in vCenter Server, an error appears: "There is no more space for virtual disk '.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session."
  • Storage vMotion fails with a pending question: msg.hbacommon.outofspace.
  • CPU Contention: VM exhibits 99.999% CPU Contention or high %WAIT in esxtop, despite low CPU demand within the guest OS.
  • Linux Guest OS Warnings: kernel:[.......] Buffer I/O error on device dm-0, logical block 500054 kernel:[.......] blk_update_request: I/O error, dev sda, sector 1213223233
  • Log Entries (vmware.log): YYYY-MM-DDTHH:MM:SS.708+0000In(05) vmx - [msg.hbacommon.outofspace] There is no more space for virtual disk '/vmfs/volumes/DATASTORE_LOCATION/VM_NAME/VM_DISK.vmdk'.
  • Log Entries (vmkernel.log): cpu24:887###3)Fil6: 4094: 'Datastore_Name': Fil6 file IO () : No space left on device
  • Error message on the UI : There is no more space for Virtual disk.

Environment

  • VMware vSphere ESXi 7.x
  • VMware vSphere ESXi 8.x
  • Thin-provisioned virtual disks
  • Linux Guest OS (NFS/NetApp environments)

Cause

The VMFS datastore has exhausted both Small File Blocks (SFBs) and Large File Blocks (LFBs). This prevents the allocation of new blocks for virtual disk writes, swap file growth, or snapshot consolidation.

Consolidation Failure Requirement: During snapshot consolidation, ESXi requires additional metadata and block-level overhead to merge delta blocks into the base disk. If the active snapshot and base disk have already consumed the available datastore capacity, there is no "working space" left to facilitate the merge operation.

Resolution

  1. Clear unwanted data, such as old log bundles and ISO images, to immediately free up space.
  2. Expand the LUN capacity from the backend storage array and increase the VMFS datastore size (e.g., expanding from 6TB to 10TB to provide overhead for large snapshots).
  3. Consolidate or delete existing virtual machine snapshots. Note that a 20% free space buffer is recommended for stable thin-provisioning and consolidation operations.
  4. Perform a Storage vMotion of the virtual machine to a datastore with more free space.
  5. Add a temporary memory reservation to the virtual machine if the swap file is too large.
  6. Check whether the datastore is a destination for vSphere Replication-based migrations (e.g., VMware HCX).
  7. Clone the virtual machine disk (VMDK) to another datastore using the vmkfstools utility.
  8. Do not attempt to restart ESXi services or reboot the ESXi server, as this will not free up datastore space.

Additional Information

Powered by Wolken Software