Virtual Machines with LWD filter attached are restarted by HA during snapshot operations.
search cancel

Virtual Machines with LWD filter attached are restarted by HA during snapshot operations.

book

Article ID: 344905

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • A virtual machine is powered off during snapshot creation.
  • The virtual machine is currently or has in the past been protected by a solution that uses the LWD filter. (VCDR, Dell PPDM, etc..)
  • Messages similar to the following are seen in the virtual machines vmware.log file located with the virtual machine files.
2022-09-14T07:25:08.060Z In(05) vcpu-0 - PluginLdr_Load: Loaded plugin 'libvmiof-disk-vmwarelwd.so' from '/usr/lib64/vmware/plugin/libvmiof-disk-vmwarelwd.so'
2022-09-14T07:25:08.060Z In(05) vcpu-0 - LWD: Initializing the LWD filter (process: 6079822)
2022-09-14T07:25:08.060Z In(05) vcpu-0 - FiltLib: VMKPrivate_FiltModInitDiskInfo failed: "Out of memory (ok to retry)" (195887125).
2022-09-14T07:25:08.060Z In(05) vcpu-0 - FiltLib: FiltLibAttachToFiltMod failed with error: "Operation failed" (1).

2022-09-14T07:25:08.060Z In(05) vcpu-0 - DISKLIB-LIB   : DiskLibFiltLibInit: Failed to create filtLib context: Operation failed (334).
2022-09-14T07:25:08.060Z In(05) vcpu-0 - DISKLIB-LIB   : DiskLibOpenInt: Failed to create filtLib context: Operation failed (334).
2022-09-14T07:25:08.061Z In(05) vcpu-0 - DISKLIB-VMFS  : "/vmfs/volumes/datastore/vm01/vm01_2-000001-sesparse.vmdk" : closed.
2022-09-14T07:25:08.062Z In(05) vcpu-0 - DISKLIB-VMFS  : "/vmfs/volumes/datastore/vm01/vm01_2-flat.vmdk" : closed.
2022-09-14T07:25:08.062Z In(05) vcpu-0 - DISKLIB-LIB   : Failed to open '/vmfs/volumes/datastore/vm01/vm01_2-000001.vmdk' with flags 0xa Operation failed (334).
2022-09-14T07:25:08.062Z Er(02) vcpu-0 - DISK: Cannot open disk '/vmfs/volumes/datastore/vm01/vm01_2-000001.vmdk': Operation failed (334).
2022-09-14T07:25:08.062Z In(05) vcpu-0 - CPT: error syncing group SCSI0
2022-09-14T07:25:08.062Z In(05) vcpu-0 - VMXNET3 user: Ethernet1 Driver Info: version = 17964782 gosBits = 2 gosType = 2, gosVer = 25344, gosMisc = 212
2022-09-14T07:25:08.062Z In(05) vcpu-0 - CPT: vm was stunned for 1160215 us
2022-09-14T07:25:08.062Z In(05) vcpu-0 - Msg_Post: Error
2022-09-14T07:25:08.062Z In(05) vcpu-0 - [msg.iofilter.failure] Operation failed
2022-09-14T07:25:08.062Z In(05) vcpu-0 - [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/datastore/vm01/vm01_2-000001.vmdk' or one of the snapshot disks it depends on.
2022-09-14T07:25:08.062Z In(05) vcpu-0 - [msg.checkpoint.continuesync.error] An operation required the virtual machine to quiesce and the virtual machine was unable to continue running.
  • At the same time, you see messages in /var/log/vmkernel.log that indicate memory exhaustion
2022-09-14T07:25:07.202Z cpu11:6079837 opID=1234eid0)6079842-UserAIOPagePool (33710422) extraMin/extraFromParent: 480/480, vm.6079822 (32700674) childEmin/eMinLimit: 57598/57819
2022-09-14T07:25:08.060Z cpu56:6079842 opID=1234eid0)filtmod-0x4317306016a0 (32700702) extraMin/extraFromParent: 11/11, vm.6079822 (32700674) childEmin/eMinLimit: 57819/57819


Environment

VMware vSphere ESXi 8.0.x
VMware vSphere ESXi 7.0.3

Cause

This issue occurs when there is no overhead memory available to load the LWD filter for the VM during snapshot creation.

Resolution

VMware is aware of this issue and working towards a fix in a future release.

Workaround:
If the affected VM(s) are not actively using the LWD filter to be protected by solutions such as VMware Cloud Disaster Recovery (VCDR) or Dell PPDM, you can work around this issue by detaching the filter from them.

To accomplish this, please see the steps and script listed in VMware KB 90283