Virtual machine stop running with error "There is no more space for virtual disk"
search cancel

Virtual machine stop running with error "There is no more space for virtual disk"

book

Article ID: 318352

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Datastore is running with low free space.
  • The virtual machine stops working.
  • If the VM is running on a snapshot, the consolidation will fail. 
  • In the summary of the virtual machine in the vCenter Server UI, there is an error message similar to:
    "There is no more space for virtual disk '<name_of_disk>.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session."
  • Also, in the summary of the virtual machine in the vCenter Server UI, an attempt to perform Storage vMotion might result in an error message similar to:
    The operation cannot be allowed at the current time because the virtual machine has a question pending: 'msg.hbacommon.outofspace:There is no more space for virtual disk '/vmfs/volumes/datastore_name/VM_name/vm.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session.'.
  • VM could be crashed with the following: /vmfs/volumes/datastore_name/VM_name/vmware.log entry.
    [DATE/TIME] In(05) vmx - [msg.hbacommon.outofspace] There is no more space for virtual disk '/vmfs/volumes/DATASTORE_LOCATION/VM_NAME/VM_DISK.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking _Retry. Click Cancel to terminate this session.
[DATE/TIME] In(05) vmx - ----------------------------------------
[DATE/TIME] In(05) vmx - MsgQuestion: msg.hbacommon.outofspace reply=1
[DATE/TIME] Cr(01) vmx - PANIC: Exiting because of failed disk operation.
  • The ESXi /var/run/log/vmkernel.log file reports “No space left on device” errors when the Virtual Machine attempted to issue I/O requests, confirming that the datastore was completely out of space:
    cpu24:887###3)Fil6: 4094: 'Datastore_Name': Fil6 file IO (<FD c41 r104>) : No space left on device
cpu24:887###3)Fil6: 4060: ioCtx: 0x45c971180f80, world: 2112600, overallStatus: No space left on device, token: 0x45c9403a9900, tokenStatus: Success, txnToken: 0x0, txnTokenStatus: N, totalIOSize: 65536, maxIOLength: 26677370880
cpu24:887###3)Fil6: 4066: numAddrs: 0, numTbz: 0, numPBsToAlloc: 0, numBlocksToAlloc: 1 numTBZBlocks: 0, numPBsToUnmap: 0, numBlocksToUnmap: 0
cpu24:887###3)Fil6: 4073: opType: 2, ioFlags: 272, allocFlags: 0, ioStateTxn: 0, ioStateNoTxn: 0, txnState: 3, numResolvedAddrMaps: 0, numTxnPendingAddrMaps: 16
cpu24:887###3)Fil6: 4078: wakeupOnIOComplete: 0, isIOIssued: 0, isIODone: 0, isRead: 0 isSync: 0, canBlock: 1, lastPartialBlock: 0
cpu24:887###3)Res3: 1874: 'Datastore_Name': [rt 1] Converted clusters updated (on-disk 30688, in-memory 30496)
cpu43:887###3)Fil3: 10596: SFB totalNumResources: 15726592 numFreeResources: 19521 numNotAvailable 14336
cpu43:887###3)Fil3: 10604: LFB totalNumResources: 30716 numFreeResources: 28 numNotAvailable 30688
  • Error message on the UI

Environment

  • VMware vSphere ESXi 7.x
  • VMware vSphere ESXi 8.x

Cause

The VMFS datastore has exhausted both Small File Blocks (SFBs) and Large File Blocks (LFBs), preventing the allocation of new blocks for virtual disk writes or for swap file growth.

Resolution

To resolve this issue, follow the steps below:

  1. Clear unwanted data, such as old log bundles and ISO images, to free up space on the datastore.
    • Check whether the datastore in question is used as a destination for any vSphere Replication-based migrations (e.g., HCX).
  2. Perform a Storage vMotion of the VM to a datastore with more free space (or one that can be resized).
  3. Consolidate or delete snapshots with the KB "An error occurred while consolidating disks" when deleting one or more snapshots or consolidate disks.
  4. If the swap file is too large for the VM, consider temporarily adding a memory reservation to the VM.
  5. Increase the LUN capacity by following the steps below, or follow Failed to expand VMFS datastore - Cannot change the host configuration.

    1. From backend storage, the datastore capacity has to be increased first.
    2. Rescan storage for ESXi from vCenter Server.
    3. Post that, from the ESXi ui, select "Increase datastore capacity" and select the option "Extent an existing extent".
    4. Make sure the increased capacity is reflected correctly.
    5. Leave a buffer space of a couple of MB's and allocate the remaining to extend the datastore. Click "Finish" to complete the extent task. 
    6. The datastore will be extended to the desired capacity on vCenter Server and ESXi.

  6. If the LUN is thin-provisioned on the storage array, ensure the pool has sufficient free capacity.
  7. Clone the vmdk to another datastore with enough free space using KB Cloning and converting virtual machine disks with vmkfstools.
  8. Restarting ESXi services or rebooting the ESXi server will not help to free up space.

Additional Information

Powered by Wolken Software