vMotion failing with error "A general system error occurred: Launch failure. Cause: Out of resource"

Products

• VMware vSAN

Issue/Introduction

• vMotion task for Virtual machine may fail with the below error.
  
  A general system error occurred: Launch failure. Cause: Out of resource.
  
  
• From the ESXi host logs, /var/run/log/vmkwarning.log may report the following event 'Failed to extend swap file'.

YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu11:70121116)WARNING: SwapExtend: 703: Current swap file size is 0 KB.
YYYY-MM-DDTHH:MM:SS Wa(180) vmkwarning: cpu11:70121116)WARNING: SwapExtend: vm 70121116: 707: Failed to extend swap file. type=regular from 0 KB to 81920 KB. currently 0. status=No space l

• From the ESXi host logs, /var/run/log/vmkernel.log may report 'No space left on device' for the object', where the UUID will be VM folder UUID.

YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu26:88136083)Fil3: 11066: Max no space retries (10) exceeded for caller Fil6_FileIOInt (status 'No space left on device')
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu26:88136083)Fil6: 4289: '########-####-####-####-############': Fil6 file IO (<FD c7 r11>) : No space left on device
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu26:88136083)Fil6: 4251: ioCtx: 0x45ba3c9f1f40, world: 88136083, overallStatus: No space left on device, token: 0x0, tokenStatus: N, txnToken: 0x0, txnTokenStatus: N, totalIOSize: 1048576, maxIOLength: 102666240

Environment

VMware vSAN

Cause

vMotion fails when the VM folder usage reached 255GB, which the maximum size for a vSAN namespace object.


For example:-
'du -h' command output from the VM folder shows the overall folder usage as 255GB and there are multiple stats folder consuming space.

[root@ESXi_host01:/vmfs/volumes/vsanDatastore/########-####-####-####-############] du -h
1.0M    ./.sdd.sf
256.0K  ./.dvsData/## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##
384.0K  ./.dvsData
1.4G    ./stats-9
2.5G    ./stats-8
32.0G   ./stats-7
60.2G   ./stats-6
62.3M   ./stats-5
156.6G  ./stats-4
557.8M  ./stats-3
2.7M    ./stats-2
3.4M    ./stats-1
64.0M   ./stats-0
156.3M  ./stats
255.0G  .       ------------->> Overall VM folder usage

Resolution

• Verify the folders/files consuming space on the VM folder.
• If these folders/files are not needed, move / delete these folders/files from the VM folder to free-up space on the vSAN namespace object.