Virtual Machines with LWD filter attached are restarted by HA during snapshot operations.
Article ID: 344905
Updated On:
Products
Issue/Introduction
Symptoms:
- A virtual machine is powered off during snapshot creation.
- The virtual machine is currently or has in the past been protected by a solution that uses the LWD filter. (VCDR, Dell PPDM, etc..)
- Messages similar to the following are seen in the virtual machines vmware.log file located with the virtual machine files.
2022-09-14T07:25:08.060Z In(05) vcpu-0 - LWD: Initializing the LWD filter (process: 6079822)
2022-09-14T07:25:08.060Z In(05) vcpu-0 - FiltLib: VMKPrivate_FiltModInitDiskInfo failed: "Out of memory (ok to retry)" (195887125).
2022-09-14T07:25:08.060Z In(05) vcpu-0 - FiltLib: FiltLibAttachToFiltMod failed with error: "Operation failed" (1).
2022-09-14T07:25:08.060Z In(05) vcpu-0 - DISKLIB-LIB : DiskLibFiltLibInit: Failed to create filtLib context: Operation failed (334).
2022-09-14T07:25:08.060Z In(05) vcpu-0 - DISKLIB-LIB : DiskLibOpenInt: Failed to create filtLib context: Operation failed (334).
2022-09-14T07:25:08.061Z In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/datastore/vm01/vm01_2-000001-sesparse.vmdk" : closed.
2022-09-14T07:25:08.062Z In(05) vcpu-0 - DISKLIB-VMFS : "/vmfs/volumes/datastore/vm01/vm01_2-flat.vmdk" : closed.
2022-09-14T07:25:08.062Z In(05) vcpu-0 - DISKLIB-LIB : Failed to open '/vmfs/volumes/datastore/vm01/vm01_2-000001.vmdk' with flags 0xa Operation failed (334).
2022-09-14T07:25:08.062Z Er(02) vcpu-0 - DISK: Cannot open disk '/vmfs/volumes/datastore/vm01/vm01_2-000001.vmdk': Operation failed (334).
2022-09-14T07:25:08.062Z In(05) vcpu-0 - CPT: error syncing group SCSI0
2022-09-14T07:25:08.062Z In(05) vcpu-0 - VMXNET3 user: Ethernet1 Driver Info: version = 17964782 gosBits = 2 gosType = 2, gosVer = 25344, gosMisc = 212
2022-09-14T07:25:08.062Z In(05) vcpu-0 - CPT: vm was stunned for 1160215 us
2022-09-14T07:25:08.062Z In(05) vcpu-0 - Msg_Post: Error
2022-09-14T07:25:08.062Z In(05) vcpu-0 - [msg.iofilter.failure] Operation failed
2022-09-14T07:25:08.062Z In(05) vcpu-0 - [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/datastore/vm01/vm01_2-000001.vmdk' or one of the snapshot disks it depends on.
2022-09-14T07:25:08.062Z In(05) vcpu-0 - [msg.checkpoint.continuesync.error] An operation required the virtual machine to quiesce and the virtual machine was unable to continue running.
- At the same time, you see messages in /var/log/vmkernel.log that indicate memory exhaustion
2022-09-14T07:25:08.060Z cpu56:6079842 opID=1234eid0)filtmod-0x4317306016a0 (32700702) extraMin/extraFromParent: 11/11, vm.6079822 (32700674) childEmin/eMinLimit: 57819/57819
Environment
VMware vSphere ESXi 7.x
Cause
Resolution
To resolve this issue if the affected VM(s) are not actively using the LWD filter to be protected by solutions such as VMware Cloud Disaster Recovery (VCDR) or Dell PPDM or Replication solutions, you can detach the filter from them.
To accomplish this, please see the steps and script listed in KB - Putting ESXi into Maintenance mode, vMotioning or reconfiguring a VM fails due to the vmwarelwd ioFilter being attached
Additional Information
Monitoring the memory usage.
# memstats -r group-stats -v -s name:parGid:eMin:eMinPeak:rMinPeak:consumed:consumedPeak -u KB | grep -i vmx
### Above command will list the vmx group ID.
### First column is VMX Group ID
# watch memstats -r group-stats -v -s name:parGid:eMin:eMinPeak:rMinPeak:consumed:consumedPeak -u KB -g <VMX group ID>
### Use the VMX Group ID from first command and replace it with <VMX group ID>
### You need to monitor consumed and consumedPeak column
### Example:
------------------------------------------------------------------------------------------------------- gid name parGid eMin eMinPeak rMinPeak consumed consumedPeak------------------------------------------------------------------------------------------------------- 363 test-vmx.v00-0x4305ff1026b0 264 1072 134788 134788 1044 134788-------------------------------------------------------------------------------------------------------
Feedback
