vMotion of virtual machines fails with Error: Failed to attach filter <scsi_filter_name>

Products

• VMware vSphere ESXi

Issue/Introduction

• Virtual machine vMotion fails between ESXi hosts.
  
  
• During vMotion, the vSphere Client reports the following error message for the virtual machine:
  

• • For 'active_drive' scsi_filter:
    The VM failed to resume on the destination during early power on. Module DevicePowerOn power on failed. Unable to create virtual SCSI device for scsi0:0, '/vmfs/volumes/datastore/virtual_machine_folder/virtual_machine.vmdk' Failed to attach filter 'active_drive' to scsi0:0: msg.vmk.status.VMK_NOT_FOUND (195887107).


  • For 'esx_splitter' scsi_filter:
    The VM failed to resume on the destination during early power on. Module DevicePowerOn power on failed. Unable to create virtual SCSI device for scsi0:0, '/vmfs/volumes/datastore/virtual_machine_folder/virtual_machine.vmdk' Failed to attach filter 'esx_splitter' to scsi0:0: msg.vmk.status.VMK_NOT_FOUND (195887107).

• The virtual machine's vmware.log will contain an error esx_splitter or active_drive, which is represented by <scsi_filter_name> in the following log snippet:
  
  YYYY-MM-DDTHH:MM:SS.957Z| vmx| I120: [msg.migrate.resume.fail] The VM failed to resume on the destination during early power on.
YYYY-MM-DDTHH:MM:SS.957Z| vmx| I120: Msg_Post: Error
YYYY-MM-DDTHH:MM:SS.957Z| vmx| I120: [msg.scsi.esx.filterAttachmentFailed] Failed to attach filter '<scsi_filter_name>' to scsi0:0: Not found (195887107).
YYYY-MM-DDTHH:MM:SS.957Z| vmx| I120: [msg.scsi.esx.virtualDevFailed] Unable to create virtual SCSI device for scsi0:0, '/vmfs/volumes/datastore/virtual_machine_folder/virtual_machine.vmdk'
YYYY-MM-DDTHH:MM:SS.957Z| vmx| I120: [msg.moduletable.powerOnFailed] Module DevicePowerOn power on failed.

• The /var/log/hostd.log will contain an error showing either esx_splitter or active_drive, which is represented by <scsi_filter_name> in the following log snippet:
  
  YYYY-MM-DDTHH:MM:SS.964Z [33999B70 verbose 'Vmsvc.vm:/vmfs/volumes/datastore/virtual_machine_folder/virtual_machine.vmx'] Handling message_vmx1: Failed to attach filter '<scsi_filter_name>' to scsi0:0: Not found (195887107).
--> Unable to create virtual SCSI device for scsi0:0, '/vmfs/volumes/datastore/virtual_machine_folder/virtual_machine.vmdk'
--> Module DevicePowerOn power on failed.
--> The VM failed to resume on the destination during early power on.
YYYY-MM-DDTHH:MM:SS.975Z [33001B70 verbose 'Vmsvc.vm:/vmfs/volumes/datastore/virtual_machine_folder/virtual_machine.vmx'] VMotionLastStatusCb: Failed with error 7: The VM failed to resume on the destination during early power on.
message = "Failed to attach filter '<scsi_filter_name>' to scsi0:0: Not found (195887107).
message = "Unable to create virtual SCSI device for scsi0:0, '/vmfs/volumes/datastore/virtual_machine_folder/virtual_machine.vmdk'
reason = "The VM failed to resume on the destination during early power on.

Environment

• VMware vSphere ESXI 7.x/8.x

Cause

This issue occurs due to a VIB mismatch, specifically when a VIB module installed on the source ESXi host is missing from the destination ESXi host.

Resolution

Removing a VIB from an ESXi host does not always automatically remove the associated entries from the virtual machine configuration (.vmx) files.

To resolve this issue, apply one of the following methods based on the number of affected virtual machines.

Method 1: If all virtual machines on the source host are affected:

• Resolve the software mismatch by installing the missing VIB on the destination ESXi host. Ensure the destination host is configured with the exact same software packages as the source ESXi host.

Method 2: If only a few virtual machines are affected: Manually remove all I/O filter entries from the configuration files (.vmx) of the affected virtual machines:

1. Power off the affected virtual machine.
2. Connect to the ESXi host via SSH using root credentials.
3. Navigate to the virtual machine directory:
   cd /vmfs/volumes/<datastore_name>/<virtual_machine_folder>/

4. Create a backup of the configuration file:
   cp virtual_machine.vmx virtual_machine.vmx.bak

5. Open the virtual machine configuration (.vmx) file using the `vi` editor:
   vi virtual_machine.vmx

6. Locate and remove/comment out (add `#` at the start of the line) ALL SCSI filter entries, not limited to `scsi0:0`.
    Look for lines matching:
   - `scsi*:*.filters = "<scsi_filter_name>"` (e.g., `scsi0:0.filters = "esx_splitter"`, `scsi0:1.filters = "esx_splitter"`)
   - `iofilter.<scsi_filter_name>.*`

7. Save the changes and exit `vi` (Press `Esc`, type `:wq!`, and press `Enter`).

8. Power on the virtual machine.

9. Proceed with migrating (vMotion) the virtual machine to the destination ESXi host.