Errors reported on vmware.log: "Out of memory (ok to retry)" and "Operation failed (334)" on taking backup of the virtual machines.
search cancel

Errors reported on vmware.log: "Out of memory (ok to retry)" and "Operation failed (334)" on taking backup of the virtual machines.

book

Article ID: 410658

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • Virtual Machines back can fail.
  • Error messages being captured on log suite related to memory issues.
  • Virtual machine is operational with no performance issues reported.

 

The following log entries are reported:


vmware.log:

YYYY-MM-DDTHH:MM:SS In(05) ####-# - FiltLib: VMKPrivate_FiltModInitDiskInfo failed: "Out of memory (ok to retry)" (#######).
YYYY-MM-DDTHH:MM:SS In(05) ####-# - FiltLib: FiltLibAttachToFiltMod failed with error: "Operation failed" (1).
YYYY-MM-DDTHH:MM:SS Er(02) ####-# - DISKLIB-LIB   : DiskLibFiltLibInit: Failed to create filtLib context: Operation failed (334).
YYYY-MM-DDTHH:MM:SS In(05) ####-# - DISKLIB-LIB   : DiskLibOpenInt: Failed to create filtLib context: Operation failed (334).
YYYY-MM-DDTHH:MM:SS In(05) ####-# - DISKLIB-LIB  : "vsan://################-################/VM_NAME" : closed.
YYYY-MM-DDTHH:MM:SS In(05) ####-# - DISKLIB-LIB   : Failed to open '/vmfs/volumes/vsan:################-################/VM_NAME/VM_NAME.vmdk' with flags 0x20a Operation failed (334).
YYYY-MM-DDTHH:MM:SS In(05) ####-# - SNAPSHOT:Failed to open disk /vmfs/volumes/vsan:################-################/VM_NAME/VM_NAME.vmdk: Operation failed (334).

 

vpxa.log:

YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103609]: [Originator@6876 sub=Default opID=########-##] [VpxLRO] -- ERROR ####-##### -- ########-####-####-####-############ -- vm-VM_ID -- vim.VirtualMachine.consolidateDisks: :vim.fault.GenericVmConfigFault
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: --> Result:
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: --> (vim.fault.GenericVmConfigFault) {
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->    faultCause = (vmodl.MethodFault) null,
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->    faultMessage = (vmodl.LocalizableMessage) [
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->       (vmodl.LocalizableMessage) {
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->          key = "msg.snapshot.vigor.consolidate.error",
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->          arg = (vmodl.KeyAnyValue) [
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->             (vmodl.KeyAnyValue) {
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->                key = "1",
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->                value = "msg.iofilter.failure"
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->             }
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->          ],
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->          message = "An error occurred while consolidating disks: Operation failed."
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->       },
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->       (vmodl.LocalizableMessage) {
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->          key = "msg.snapshot.onlineconsolidate.failed",
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->          arg = (vmodl.KeyAnyValue) [
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->             (vmodl.KeyAnyValue) {
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->                key = "1",
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->                value = "scsi0:0"
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->             },
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->             (vmodl.KeyAnyValue) {
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->                key = "2",
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->                value = "msg.iofilter.failure"
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->             }
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->          ],
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->          message = "Consolidation failed for disk node 'scsi0:0': Operation failed."
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->       }
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->    ],
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->    reason = "An error occurred while consolidating disks: Operation failed."
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->    msg = "An error occurred while consolidating disks: Operation failed."
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: --> }
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: --> Args:
YYYY-MM-DDTHH:MM:SS Er(163) Vpxa[2103571]: -->

 

vmkernel.log:

YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu90:vm.VM_ID)Sched: vm vm.VM_ID: 6621: renamed group ######## to vm.vm.VM_ID
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu90:vm.VM_ID)Sched: vm vm.VM_ID: 6637: group ######## is located under group 40981
...
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu107:10873431)Admission failure in path: host/user/pool0/vm.vm.VM_ID:filtmod.vm.VM_ID
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu107:10873431)filtmod.vm.VM_ID (#######) requires 60 KB, asked 60 KB from vm.vm.VM_ID (########) which has 160352 KB occupied and 20 KB available.
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu107:10873431)Admission failure in path: host/user/pool0/vm.vm.VM_ID:filtmod.vm.VM_ID
YYYY-MM-DDTHH:MM:SS In(182) vmkernel: cpu107:10873431)filtmod.vm.VM_ID (#######) requires 60 KB, asked 60 KB from vm.vm.VM_ID (########) which has 160352 KB occupied and 20 KB available.

 

 

 

Environment

ESXi hosts 8.0

Cause

The virtual machine is under memory pressure (even though it is not presenting memory issues). 
For instance in the logs above as per vmkernel indicates, vm consumed memory is 16Gb, VM itself is consuming all the memory so filtmod can not get memory free for the iofilter.

Resolution

Increase the memory for the affected virtual machine and set a memory reservation for it.

Powered by Wolken Software