vMotion / DRS fails with error "ThrowabIeProxy.cause Out of resources"

Products

• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

vMotion or DRS fails to migrate VMs, with the following failure message:
A general system error occurred: Launch failure ####-##-##T##:## Out of resources ThrowabIeProxy.cause Out of resources ####-##-##T##:## 

Snapshot creation fails, causing VM snapshot-based backup failures, with the following failure message:
An error occurred while saving the snapshot: msg.changetracker.CREATEMlRROR. An error occurred while taking a snapshot: msg.changetracker.CREATEMlRROR. 

VMs that are powered off during an OS patch activity do not power back on with the following message:
swap file cannot be created on vVOL

Environment

VMware vSphere ESXi 7.0

VMware vSphere ESXi 8.0

Cause

When a vvol-based VM is created, vCenter creates a 4GB thin-provisioned configuration vvol (config vvol) on the array, which ESXi formats with VMFS. A VM’s config vvol stores the files required to build and manage it: its VMX file, logs, VMDK pointers, etc.

Config vvol has a finite size of 4GB. If it fills up, the aforementioned symptoms are observed. More information is available at Understanding Virtual Volumes (vVols) in VMware vSphere 6.7/7.0/8.0


Related logs and observations:

hostd log:

####-##-##T##:## verbose hostd[6022036] [Originator@6876 sub=Vigor.Vmsvc.vm:/vmfs/volumes/vvol:eef43cb3c45f35e1-################/rfc####.########-fb5b-48ad-a20a-77e7f78f7c92/VM_Name.vmx] Create Snapshot message: An error occurred while saving the snapshot: Could not create mirror of change tracking file.
--> An error occurred while taking a snapshot: Could not create mirror of change tracking file.
-->
..
..
####-##-##T##:## info hostd[6022053] [Originator@6876 sub=Vimsvc.TaskManager] Task Completed : haTask-##-vim.VirtualMachine.createSnapshot-####### Status error

vmkernel log:

####-##-##T##:## cpu57:3565803)WARNING: Res3: 7549: 'rfc####.########-fb5b-48ad-a20a-77e7f78f7c92': [rt 1] No Space - did not find enough resources after second pass! (needed: 1, found: 0)
####-##-##T##:## cpu57:3565803)Res3: 7550: 'rfc####.########-fb5b-48ad-a20a-77e7f78f7c92': [rt 1] Resources t 0, e 0, np 200, wb 0, b 0, RCs u 0, i 0, nf 20, pe 0, oe 0

For VMs failing to power on, you will find the following in vmkernel log:

####-##-##T##:## cpu13:2161040)WARNING: SwapExtend: 715: Failed to extend swap file /vmfs/volumes/vvol:19c3ee3262253c73--################/rfc####.########-716d-43df-b170-7cb32ee3ceaf/vmx-dc6-xterm-40-f3acb5e09376d10ef8e44620ed61ae27b3d407$
####-##-##T##:## cpu13:2161040)WARNING: SwapExtend: 718: Current swap file size is 0 KB.
####-##-##T##:## cpu13:2161040)WARNING: SwapExtend: vm 2161040: 726: Failed to extend swap file. type=regular from 0 KB to 81920 KB. currently 0. status=No space left on device
####-##-##T##:## cpu13:2161040)MemSched: 7179: Could not set prereserve swap space.
####-##-##T##:## cpu13:2161040)WARNING: Res3: 7548: 'rfc####.########-716d-43df-b170-7cb32ee3ceaf': [rt 1] No Space - did not find enough resources after second pass! (needed: 9, found: 0)
####-##-##T##:## cpu13:2161040)Res3: 7549: 'rfc####.########-716d-43df-b170-7cb32ee3ceaf': [rt 1] Resources t 0, e 0, np 199, wb 0, b 0, RCs u 0, i 0, nf 20, pe 0, oe 0

Config vvol of the VM has run out of space:

[root@##########] vmkfstools -Ph -v10 /vmfs/volumes/vvol:eef43cb3c45f35e1-################/rfc####.########-fb5b-48ad-a20a-#######/
VMFS-5.81 (Raw Major Version: 14) file system spanning 1 partitions.
File system label (if any): rfc####.########-fb5b-48ad-a20a-77e7f78f7c92
Mode: public ATS-only
Capacity 4 GB, 0 bytes available, file block size 1 MB, max supported file size 62.9 TB
Volume Creation Time: Tue Apr ## ##:##:## ####
Files (max/free): 40704/40677
Ptr Blocks (max/free): 64512/64492
Sub Blocks (max/free): 16384/16379
Secondary Ptr Blocks (max/free): 256/256
File Blocks (overcommit/used/overcommit %): 0/4096/0
Ptr Blocks  (overcommit/used/overcommit %): 0/20/0
Sub Blocks  (overcommit/used/overcommit %): 0/5/0
Volume Metadata size: 492175360
Disk Block Size: 512/512/0
UUID: ########-91123296-7c8b-############
Logical device:
Partitions spanned (on "vvol"):
        rfc####.########-fb5b-48ad-a20a-77e7f78f7c92
Is Native Snapshot Capable: YES
OBJLIB-LIB: ObjLib cleanup done.
WORKER: asyncOps=0 maxActiveOps=0 maxPending=0 maxCompleted=0

In this case, we find that the current or rotated vmware.log files are large in size and make up for majority of the usage of the config vvol:

[root@##########] ls -lah /vmfs/volumes/vvol:eef43cb3c45f35e1-################/rfc####.########-fb5b-48ad-a20a-##########/
..
..
drwxrwsr-x  2  790 Mar 10 13:45  .
drwxrwsr-x 25 1.4K Mar 10 13:44  ..
-rw-rw-r--  1   14 Feb  5 13:06  VM_Name-aux.xml
-rw-rw-r--  1  736 Feb  5 13:06  VM_Name.vmdk
-rw-rw-r--  1   45 Feb  5 13:06  VM_Name.vmsd
-rw-rw-r--  1 5.1K Mar 10 09:51  VM_Name.vmx
-rw-rw-r--  1   47 Apr 15  2023  VM_Name.vmxf
-rw-rw-r--  1  684 Feb  6 04:07  VM_Name_1.vmdk
..
..
-rw-rw-r--  1 1.2G Apr 13  2023  vmware.log
-rw-rw-r--  1 478M Apr 13  2023  vmware-0.log
-rw-rw-r--  1  99M Jul 21  2023  vmware-57.log
-rw-rw-r--  1 176M Sep  8  2023  vmware-58.log
-rw-rw-r--  1 298K Sep  8  2023  vmware-59.log
-rw-rw-r--  1  13M Sep 11  2023  vmware-60.log
-rw-rw-r--  1 1.1G Apr 10  2024  vmware-61.log
-rw-rw-r--  1 352M Jun 21  2024  vmware-62.log

In other cases, large VM crash dump files and even manually created files were identified, in this directory.

Resolution

Remove the identified unusually large files (vmware.log, filename.dmp and other files that were created manually) from the config vvol of the VM (VM directory), using the vCenter or over a SSH session to the vCenter.