Admission failure in path: host/user/pool1/vm<ID>filtmod<ID>
search cancel

Admission failure in path: host/user/pool1/vm<ID>filtmod<ID>

book

Article ID: 419091

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Virtual Machine abruptly powers off with the below symptoms:

  • vmkernel.log reports the following events indicating that the IOFilter module was unable to allocate memory.

    YYYY-MM-DDTHH:MM:SS.245Z In(182) vmkernel: cpu86:13263448 opID=ce748739)Admission failure in path: host/user/pool1/vm.13263417:filtmod.13263417
YYYY-MM-DDTHH:MM:SS.245Z In(182) vmkernel: cpu86:13263448 opID=ce748739)filtmod.13263417 (63092462) requires 8 KB, asked 8 KB from vm.13263417 (63092434) which has 234428 KB occupied and 0 KB available.
YYYY-MM-DDTHH:MM:SS.245Z In(182) vmkernel: cpu86:13263448 opID=ce748739)Admission failure in path: host/user/pool1/vm.13263417:filtmod.13263417
YYYY-MM-DDTHH:MM:SS.245Z In(182) vmkernel: cpu86:13263448 opID=ce748739)filtmod.13263417 (63092462) requires 8 KB, asked 8 KB from vm.13263417 (63092434) which has 234428 KB occupied and 0 KB available.

     

  • The virtual machine vmware.log reports the below events.

    YYYY-MM-DDTHH:MM:SS.245Z In(05) vcpu-0 73cb8e0d LWD: Initializing the LWD filter (process: 13263417)
YYYY-MM-DDTHH:MM:SS.245Z In(05) vcpu-0 73cb8e0d FiltLib: VMKPrivate_FiltModInitDiskInfo failed: "Out of memory (ok to retry)" (195887125).
YYYY-MM-DDTHH:MM:SS.245Z In(05) vcpu-0 73cb8e0d FiltLib: FiltLibAttachToFiltMod failed with error: "Operation failed" (1).
YYYY-MM-DDTHH:MM:SS.245Z Er(02) vcpu-0 73cb8e0d DISKLIB-LIB   : DiskLibFiltLibInit: Failed to create filtLib context: Operation failed (334).
YYYY-MM-DDTHH:MM:SS.245Z In(05) vcpu-0 73cb8e0d DISKLIB-LIB   : DiskLibOpenInt: Failed to create filtLib context: Operation failed (334).
YYYY-MM-DDTHH:MM:SS.245Z In(05) vcpu-0 73cb8e0d DISKLIB-VMFS  : "/vmfs/volumes/########-########-1e52-############/<VM_NAME>/<VM_NAME>-flat.vmdk" : closed.
YYYY-MM-DDTHH:MM:SS.245Z In(05) vcpu-0 73cb8e0d DISKLIB-LIB   : Failed to open '/vmfs/volumes/########-########-1e52-############/<VM_NAME>/<VM_NAME>.vmdk' with flags 0xa Operation failed (334).
YYYY-MM-DDTHH:MM:SS.245Z Er(02) vcpu-0 73cb8e0d DISK: Cannot open disk '/vmfs/volumes/########-########-1e52-############/<VM_NAME>/<VM_NAME>.vmdk': Operation failed (334)
...
YYYY-MM-DDTHH:MM:SS.246Z In(05) vcpu-0 73cb8e0d CPT: error syncing group SCSI0
YYYY-MM-DDTHH:MM:SS.246Z In(05) vcpu-0 73cb8e0d CPT: vm was stunned for 377503 us
YYYY-MM-DDTHH:MM:SS.246Z In(05) vcpu-0 73cb8e0d Msg_Post: Error
YYYY-MM-DDTHH:MM:SS.246Z In(05) vcpu-0 73cb8e0d [msg.iofilter.failure] Operation failed
YYYY-MM-DDTHH:MM:SS.246Z In(05) vcpu-0 73cb8e0d [msg.disk.noBackEnd] Cannot open the disk '//vmfs/volumes/########-########-1e52-############/<VM_NAME>/<VM_NAME>.vmdk' or one of the snapshot disks it depends on.
YYYY-MM-DDTHH:MM:SS.246Z In(05) vcpu-0 73cb8e0d [msg.checkpoint.continuesync.error] An operation required the virtual machine to quiesce and the virtual machine was unable to continue running.

Environment

VMware ESXi 7.x
VMware ESXi 8.x
VMware vCenter 7.x
VMware vCenter 8.x

Cause

When an IOFilter is attached to the VMs disk, some overhead memory is required for the reconfigure operation to succeed.
DRS calculates the required amount of memory overhead and sets the limit at a per VM level. In rare circumstances, the LWD filter reconfigure operations requests more memory than the set limit. This request for additional memory can result in a VM crash. 

Resolution

A fix will be made available in a future release of vSphere.

To mitigate the issue from recurring, increase VM overhead memory.

To increase the MemOverheadGrowth at a cluster level, follow the below steps:

  • SSH to the vCenter Sever as the root user.
  • Create a copy of the file /etc/vmware-vpx/vpxd.cfg. The copied vpxd.cfg file can be used to roll back any changes.

    cp /etc/vmware-vpxd/vpxd.cfg /etc/vmware-vpxd/vpxd.cfg.bak

  • Edit the vpxd.cfg file using the command:

    vi /etc/vmware-vpx/vpxd.cfg

  • Add the below configuration between the <vpxd> and the </vpxd> tags:
             <cluster>
         <VMOverheadGrowthLimit>6</VMOverheadGrowthLimit>
        </cluster>


  • Save the file using the command,  esc > wq!
  • Restart the vCenter Services for changes to be pushed down to all clusters in the vCenter Server.

    service-control --stop vmware-vpxd && service-control --start vmware-vpxd

  • Restart the VMs that use the IOFilter. If VMs cannot be restarted, the VMX process of the VM can be restarted for the changes to take effect.
    Restarting the vmx process of the VM will not restart the VM. 

    As a precaution, validate the below steps in a non-critical VM before proceeding further:
    • SSH to the ESXi host using the root user account.
    • Make a note of the VMID of the VM that you want to restart the vmx process for using the below command.

      vim-cmd vmsvc/getallvms

    • Restart the vmx process of the VM using the VMID from the previous command.

      vim-cmd vmsvc/power.suspendResume <VMID>

Note: Setting VMOverheadGrowthLimit=6 configures 6*5=30MB of overhead memory. This is sufficient memory overhead at a per VM level.

Additional Information

Attaching LWD i/o filter to virtual machines fails with "SPIF filters change failure: Operation failed."

Powered by Wolken Software