"Module 'Disk' power on failed" / "Plugin loading error" when attempting to power on a VM

Products

VMware vSphere ESXi

Issue/Introduction

Virtual machine will fail to power on after removing an IO Provider VIB from the managing ESXi host
Attempting to Powering ON the VM fails with an error similar to the below.
```
"Module 'Disk' power on failed" / "Plugin loading error"
```

From the /var/run/log/hostd.log file:

Power on failed: (vim.fault.GenericVmConfigFault) {
   faultCause = (vmodl.MethodFault) null,
   faultMessage = (vmodl.LocalizableMessage) [
      (vmodl.LocalizableMessage) {
         key = "msg.iofilter.pluginError",
         arg = <unset>,
         message = "Plugin loading error"
      },
      (vmodl.LocalizableMessage) {
         key = "msg.disk.noBackEnd",
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               key = "1",
               value = "/vmfs/volumes/<datastore_id>/<vm_folder>/VMNAME_1.vmdk"
            }
         ],
         message = "Cannot open the disk '/vmfs/volumes/<datastore_id>/<vm_folder>/VMNAME_1.vmdk' or one of the snapshot disks it depends on. "
      },
      (vmodl.LocalizableMessage) {
         key = "msg.moduletable.powerOnFailed",
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               key = "1",
               value = "Disk"
            }
         ],
         message = "Module 'Disk' power on failed. "
      },
      (vmodl.LocalizableMessage) {
         key = "msg.iofilter.pluginError",
         arg = <unset>,
         message = "Plugin loading error"
      },
      (vmodl.LocalizableMessage) {
         key = "msg.disk.noBackEnd",
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               key = "1",
               value = "/vmfs/volumes/<datastore_id>/<vm_folder>/VMNAME.vmdk"
            }
         ],
         message = "Cannot open the disk '/vmfs/volumes/<datastore_id>/<vm_folder>/VMNAME.vmdk' or one of the snapshot disks it depends on. "
      },
      (vmodl.LocalizableMessage) {
         key = "msg.vmx.poweron.failed",
         arg = <unset>,
         message = "Failed to start the virtual machine."
      }
   ],
   reason = "Plugin loading error"
   msg = "Plugin loading error"
}

When we review the vmdk descriptor files for the vmdk's in the error above, they contain references to the removed IO Filter provider as shown in below example:

db.adapterType = "lslogic"
db.deletable = "true"
db.geometry.cylinders = "######"
db.geometry.heds = "##"
db.geometry.sectors = "##"
db.iofilters = "<iofilter_name>"   <--- HERE
db.logContentID = "###########"
db.sidecars = "<iofilter>, <VMNAME_#-#####>.vmdk"    <--- HERE
db.thinProvisioned = "#"
db.toolsInstallType = "#"
db.toolsVersion = "#####"
db.uuid = ## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ## "
db.virtualHWVersion = "##"

Environment

vSphere 7.x , 8.x

Cause

This issue occurs when references of the IO Provider are still present in the VM's VMDK descriptor file even after the IO filter has been removed

Resolution

To resolve this issue , remove IO filter provider from the vmdk descriptor files.

Using "vi" edit command, Remove the IO filter provider line entries from the VMDK descriptor files for virtual machines with more than one VMDK attached

After Edit

db.adapterType = "lslogic"
db.deletable = "true"
db.geometry.cylinders = "294084"
db.geometry.heds = "255"
db.geometry.sectors = "63"
db.logContentID = "208e80f99f756c948df2da1c1ca6b10"
db.thinProvisioned = "1"
db.toolsInstallTpe = "1"
db.toolsVersion = "12389"
db.uuid = "## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##"
db.vertualHWVersion = "4"

Note: DO NOT edit the descriptor without having a backup of it first. Copy the descriptor file to a safe location prior to editing.

Once edited, reload the VM to inventory using ESXi CLI (see KB 343248 for detailed steps).
```
vim-cmd vmsvc/reload VMid
```
Power on the VM from ESXi CLI (see KB 345049 for detailed steps)
```
vim-cmd vmsvc/power.on VMid
```

In the vCenter UI, confirm that all actions against the VM are working (power on-off / vMotion, etc.).