Virtual machine stop running with error message "There is no more space for virtual disk"
Article ID: 318352
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
- Datastore running with low free space.
- Virtual machine stops working.
- If the VM is running on snapshot, the consolidation will fail.
- In the summary of the virtual machine in vCenter Server UI there is an error message similar to:
"There is no more space for virtual disk '<name_of_disk>.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session."
- Also in the summary of the virtual machine in vCenter Server UI when trying to perform svmotion might be error message similar to:
The operation cannot be allowed at the current time because the virtual machine has a question pending: 'msg.hbacommon.outofspace:There is no more space for virtual disk '/vmfs/volumes/datastore_name/VM_name/vm.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session.'. - VM could be crashed with below log entry and Backtrace.
[DATE/TIME] In(05) vmx - [msg.hbacommon.outofspace] There is no more space for virtual disk '/vmfs/volumes/DATASTORE_LOCATION/VM_NAME/VM_DISK.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking _Retry. Click Cancel to terminate this session. [DATE/TIME] In(05) vmx - ---------------------------------------- [DATE/TIME] In(05) vmx - MsgQuestion: msg.hbacommon.outofspace reply=1 [DATE/TIME] Cr(01) vmx - PANIC: Exiting because of failed disk operation. [DATE/TIME] In(05) vmx - Backtrace: [DATE/TIME] In(05) vmx - Backtrace[0] 0000033d09eee500 rip=00000009cc990b69 rbx=0000033d09eee520 rbp=0000033d09eeea00 r12=00000009cdc20080 r13=0000000000000001 r14=00000009cd5d2ee8 r15=00000009cd5d2f04 [DATE/TIME] In(05) vmx - Backtrace[1] 0000033d09eeea10 rip=00000009ccb183cf rbx=00000009ce5619e0 rbp=0000033d09eeea50 r12=00000009cd93dc00 r13=00000009ce5619e5 r14=00000009cd5d2ee8 r15=00000009cd5d2f04 - The /var/run/log/vmkernel.log file reports “No space left on device” errors when Virtual Machine attempted to issue I/O requests confirming that the datastore ran completely out of space:
cpu24:8875503)Fil6: 4094: 'Datastore_Name': Fil6 file IO (<FD c41 r104>) : No space left on device cpu24:8875503)Fil6: 4060: ioCtx: 0x45c971180f80, world: 2112600, overallStatus: No space left on device, token: 0x45c9403a9900, tokenStatus: Success, txnToken: 0x0, txnTokenStatus: N, totalIOSize: 65536, maxIOLength: 26677370880 cpu24:8875503)Fil6: 4066: numAddrs: 0, numTbz: 0, numPBsToAlloc: 0, numBlocksToAlloc: 1 numTBZBlocks: 0, numPBsToUnmap: 0, numBlocksToUnmap: 0 cpu24:8875503)Fil6: 4073: opType: 2, ioFlags: 272, allocFlags: 0, ioStateTxn: 0, ioStateNoTxn: 0, txnState: 3, numResolvedAddrMaps: 0, numTxnPendingAddrMaps: 16 cpu24:8875503)Fil6: 4078: wakeupOnIOComplete: 0, isIOIssued: 0, isIODone: 0, isRead: 0 isSync: 0, canBlock: 1, lastPartialBlock: 0 cpu24:8875503)Res3: 1874: 'Datastore_Name': [rt 1] Converted clusters updated (on-disk 30688, in-memory 30496) cpu43:8875503)Fil3: 10596: SFB totalNumResources: 15726592 numFreeResources: 19521 numNotAvailable 14336 cpu43:8875503)Fil3: 10604: LFB totalNumResources: 30716 numFreeResources: 28 numNotAvailable 30688 - Error message on the UI
Environment
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x
Cause
The VMFS datastore has exhausted both Small File Blocks (SFBs) and Large File Blocks (LFBs), preventing the allocation of new blocks for virtual disk writes or swap file growth.
Resolution
To resolve this issue, follow the steps below:
- Clear unwanted data like old log bundle, ISO images, etc. to free space on the Datastore+.
- Check if the datastore in question is used as destination for any vSphere Replication based migrations (Ex: HCX).
- Consider doing a Storage VMotion of the VM to a datastore that has more free space (or that can be increased in size).
- Consolidate or delete snapshots with the KB "An error occurred while consolidating disks" when deleting one or more snapshots or consolidate disks.
- If the swap file is too large for the VM, consider temporarily adding a memory reservation to the VM.
- Consider increasing the space of the actual LUN following below steps or follow KB321398.
- From backend storage, the datastore capacity has to be increased first.
- Rescan storage for ESXi from vCenter Server.
- Post that, from ESXi ui, select "Increase datastore capacity" and select the option "Extent an existing extent".
- Make sure the increased capacity is reflecting correctly.
- Leave a buffer space of couple of MB's and allocate the remaining to extend the datastore. Click "Finish" to complete the extent task.
- The datastore will be extended to the desired capacity on vCenter Server and ESXi.
- If the LUN is thin provisioned from the storage array, ensure the pool has enough free capacity.
- Clone the vmdk to other datastore with enough free space with KB Cloning and converting virtual machine disks with vmkfstools.
- Restarting ESXi services or rebooting the ESXi server will not help to free-up space.
Additional Information
Feedback
Yes
No
Powered by
