Attaching LWD i/o filter to virtual machines fails with ""SPIF filters change failure: Operation failed.""

Products

• VMware vCenter Server 6.0
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

When performing VM backups or configuring disaster recovery (DR) protection with VMware Cloud Disaster Recovery (VCDR), LWD I/O filters are attached to the VM as part of the process. However, during filter initialization, errors such as "out of memory" can occur, which sometimes cause the virtual machine to experience a panic or crash.

Symptoms:

Log file : vmware.log

YYYY-MM-DDTHH:MM:SS.xxxZ In(05) vcpu-0 - DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = xxxxxxxxx, extentType = 2
YYYY-MM-DDTHH:MM:SS.xxxZ In(05) vcpu-0 - DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = xxxxxxxxx, extentType = 1
YYYY-MM-DDTHH:MM:SS.xxxZ In(05) vcpu-0 - PluginLdr_Load: Loaded plugin 'libvmiof-disk-vmwarelwd.so' from '/usr/lib64/vmware/plugin/libvmiof-disk-vmwarelwd.so'
YYYY-MM-DDTHH:MM:SS.xxxZ In(05) vcpu-0 - LWD: Initializing the LWD filter (process: 19146368)
YYYY-MM-DDTHH:MM:SS.xxxZ In(05) vcpu-0 - FiltLib: VMKPrivate_FiltModInitDiskInfo failed: "Out of memory (ok to retry)" (195887125).
YYYY-MM-DDTHH:MM:SS.xxxZ In(05) vcpu-0 - FiltLib: FiltLibAttachToFiltMod failed with error: "Operation failed" (1).
YYYY-MM-DDTHH:MM:SS.xxxZ In(05) vcpu-0 - DISKLIB-LIB : DiskLibFiltLibInit: Failed to create filtLib context: Operation failed (334).
YYYY-MM-DDTHH:MM:SS.xxxZ In(05) vcpu-0 - DISKLIB-LIB : DiskLibOpenInt: Failed to create filtLib context: Operation failed (334).

 

Log file : vpxa.log

YYYY-MM-DDTHH:MM:SS.xxxZ info vpxa[xxxxxxx] [Originator@6876 sub=Default opID=xxxxxxxx-xxxx-xx-xx-xx] [VpxLRO] -- ERROR task-xxxxxx -- vm-48 -- vim.VirtualMachine.reconfigure: vim.fault.GenericVmConfigFault:
--> Result:
--> (vim.fault.GenericVmConfigFault) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = (vmodl.LocalizableMessage) [
--> (vmodl.LocalizableMessage) {
--> key = "msg.disk.spifChangeFailure",
--> arg = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> key = "1",
--> value = "msg.iofilter.failure"
--> }
--> ],
--> message = "SPIF filters change failure: Operation failed."
--> }
--> ],
--> reason = "SPIF filters change failure: Operation failed."
--> msg = "SPIF filters change failure: Operation failed.

Environment

VMware ESXi 6.7x
VMware ESXi 7.x
VMware ESXi 8.x
VMware vCenter 6.7x
VMware vCenter 7.x
VMware vCenter 8.x

Cause

DRS keeps a cap on the memory overhead of the virtual machine, when an LWD filter is attached to the VM, a maximum overhead of ~20MB is needed for the reconfigure operation to succeed, However, for a DRS enabled cluster with default settings for "MemOverheadGrowth" option, the memLimit increase on host spec sync is not sufficient to perform a LWD filter reconfigure operation which results in sporadic failures.

Resolution

To increase the MemOverheadGrowth at cluster level, please follow the below steps.

1) Login to vCenter as root
2) vi /etc/vmware-vpx/vpxd.cfg (make a copy of the vpxd.cfg before modifying)
3) Add this configuration between the <vpxd> and the </vpxd> tags:
     <cluster>
     <VMOverheadGrowthLimit>6</VMOverheadGrowthLimit>
   </cluster>
4) Save the file esc > wq!
5) Restart the vCenter Services for changes to push down to all clusters in the Virtual Center
service-control --stop vmware-vpxd && service-control --start vmware-vpxd

6) Restart vmx of the VMs for thich the LWD failed

   - Check the VMID of the VM in an ssh session to the ESXi host the VM is registered to
     vim-cmd vmsvc/getallvms

   - Restart vmx of the VMs for which LWD config failed
     vim-cmd vmsvc/power.suspendResume <VMID>

   Or
   - Restart the VM if this is acceptable
7) Re-attempt attaching the LWD filter 

Note : Setting VMOverheadGrowthLimit=6 means 6*5=30MB