Storage vMotion and VM power on fails with error "No space left on device" on VMFS6 datastore

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

Storage Vmotion and VM power process fails with error "No space left on device. " on VMFS6  datastore even datastore having plenty of free space.

 

Storage vMotion Log  messages:

YYYY-MM-DDTHH:MM:SS.SSSSZ cpu23:2920025)Fil3: 10493: LFB totalNumResources: 10240 numFreeResources: 7600 numNotAvailable 2624 <-----------------------------------------------------------------
YYYY-MM-DDTHH:MM:SS.SSSSZ cpu12:2920025)Fil3: 10472: Max no space retries (10) exceeded for caller Fil3_UpdateBlocks (status 'No space left on device')
YYYY-MM-DDTHH:MM:SS.SSSSZ cpu12:2920025)FS3DM: 956: Failed to allocate 1 blocks at offset 1313653915648: No space left on device
YYYY-MM-DDTHH:MM:SS.SSSSZ cpu12:2920025)FS3DM: 2321: status No space left on device copying 1 extents between two files, bytesTransferred = 0 extentsTransferred: 0
YYYY-MM-DDTHH:MM:SS.SSSSZ cpu12:2920025)WARNING: SVM: 2867: scsi0:1 Failed SVMFDSIoctlMoveData: No space left on device <----------------------------------------------------------------------------
vpxa.log:--> value = "No space left on device"
vpxa.log:--> message = "Failed to copy source (/vmfs/volumes/########-####-########781/VM/myvm_1.vmdk) to destination (/vmfs/volumes/########-####-########6b21/VM/myvm_1.vmdk): No space left on device. "

vmkernel.log

 

YYYY-MM-DDTHH:MM:SS.SSSSZ cpu13:2142207)Fil3: 10470: SFB totalNumResources: 9023232 numFreeResources: 3488508 numNotAvailable 0
YYYY-MM-DDTHH:MM:SS.SSSSZ cpu13:2142207)Fil3: 10478: LFB totalNumResources: 17624 numFreeResources: 0 numNotAvailable 17624
YYYY-MM-DDTHH:MM:SS.SSSSZ cpu13:2142207)Fil3: 10457: Max no space retries (10) exceeded for caller Fil3_SetFileLength (status 'No space left on device')
YYYY-MM-DDTHH:MM:SS.SSSSZ cpu13:2142207)WARNING: SwapExtend: 718: Failed to extend swap file /vmfs/volumes/########-####-########6b21/VM/myvm_1.vswp from 0 KB to 67108864 KB : No space left on device

Environment

VMware vSphere ESXi 7.0.2
VMware vSphere ESXi 7.0.3

Cause

 VMFS uses different resource allocations ( such as "small file blocks" (SFBs) and "large file blocks" (LFBs)) internally for space allocation to different kind of files.

This issue occurs when all the "small file blocks" (SFBs) / "large file blocks" (LFB) are used up and we need to convert "large file blocks" (LFBs) into SFBs and Small File Block (SFBs) into LFBs.

 Due to known issue, the freshly converted SFBs are not immediately available for allocation causing "No Space" issue.

Resolution

Upgrade to vSphere ESXi version 7.0 U3c to fix the reported issue

Additional Information

If the number of numNotAvailable for both SFB and LFB is not 0,  that means a real space issue. Please expand or free the space of the datastore.