Attaching LWD i/o filter to virtual machines fails with ""SPIF filters change failure: Operation failed.""

Products

• VMware vCenter Server 6.0
• VMware vCenter Server 7.0
• VMware vCenter Server 8.0

Issue/Introduction

When performing VM backups or configuring disaster recovery (DR) protection with VMware Cloud Disaster Recovery (VCDR), LWD I/O filters are attached to the VM as part of the process. However, during filter initialization, errors such as "out of memory" can occur, which sometimes cause the virtual machine to experience a panic or crash.

Symptoms:

vmware.log

2024-08-06T22:30:34.160Z In(05) vcpu-0 - DISKLIB-CHAINESX : ChainESXOpenSubChain:(0) fid = 605511985, extentType = 2
2024-08-06T22:30:34.160Z In(05) vcpu-0 - DISKLIB-CHAINESX : ChainESXOpenSubChain:(1) fid = 220160304, extentType = 1
2024-08-06T22:30:34.161Z In(05) vcpu-0 - PluginLdr_Load: Loaded plugin 'libvmiof-disk-vmwarelwd.so' from '/usr/lib64/vmware/plugin/libvmiof-disk-vmwarelwd.so'
2024-08-06T22:30:34.161Z In(05) vcpu-0 - LWD: Initializing the LWD filter (process: 19146368)
2024-08-06T22:30:34.162Z In(05) vcpu-0 - FiltLib: VMKPrivate_FiltModInitDiskInfo failed: "Out of memory (ok to retry)" (195887125).
2024-08-06T22:30:34.162Z In(05) vcpu-0 - FiltLib: FiltLibAttachToFiltMod failed with error: "Operation failed" (1).
2024-08-06T22:30:34.162Z In(05) vcpu-0 - DISKLIB-LIB : DiskLibFiltLibInit: Failed to create filtLib context: Operation failed (334).
2024-08-06T22:30:34.162Z In(05) vcpu-0 - DISKLIB-LIB : DiskLibOpenInt: Failed to create filtLib context: Operation failed (334).

vpxa.log

2024-08-25T09:28:23.479Z info vpxa[2101597] [Originator@6876 sub=Default opID=0cf82913-2917-35-01-f7] [VpxLRO] -- ERROR task-470459 -- vm-48 -- vim.VirtualMachine.reconfigure: vim.fault.GenericVmConfigFault:
--> Result:
--> (vim.fault.GenericVmConfigFault) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = (vmodl.LocalizableMessage) [
--> (vmodl.LocalizableMessage) {
--> key = "msg.disk.spifChangeFailure",
--> arg = (vmodl.KeyAnyValue) [
--> (vmodl.KeyAnyValue) {
--> key = "1",
--> value = "msg.iofilter.failure"
--> }
--> ],
--> message = "SPIF filters change failure: Operation failed."
--> }
--> ],
--> reason = "SPIF filters change failure: Operation failed."
--> msg = "SPIF filters change failure: Operation failed.

Environment

VMware Esxi 6.7x
VMware Esxi 7.x
VMware Esxi 8.x
VMware vCenter 6.7x
VMware vCenter 7.x
VMware vCenter 8

Cause

DRS keeps a cap on the memory overhead of the virtual machine, when an LWD filter is attached to the VM,  a maximum overhead of ~20MB is needed for the reconfigure operation to succeed, However, for a DRS enabled cluster with default settings for "MemOverheadGrowth" option, the memLimit increase on host spec sync is not sufficient to perform an
LWD filter reconfigure operation which results in sporadic failures.

Resolution

To increase the MemOverheadGrowth at cluster level, please follow the below steps.

1) Login to vCenter as root
2) vi /etc/vmware-vpx/vpxd.cfg (make a copy of the vpxd.cfg before modifying)
3) add this configuration between the <vpxd> and the </vpxd> tags:
     <cluster>
     <VMOverheadGrowthLimit>6</VMOverheadGrowthLimit>
      </cluster>
4) Save the file wq!
5) restart the vCenter Services for changes to take affect 

Note: Setting VMOverheadGrowthLimit=6 means 6*5=30MB