"Module 'Disk' power on failed" / "Plugin loading error" when attempting to power on a VM

Products

• VMware vSphere ESXi
• VMware vCenter Server 8.0

Issue/Introduction

A VM will fail to power on after removing an IO Provider VIB from the managing ESXi host

Environment

ESXi 8.x

vCenter Server 8.x

Cause

References to the IO Provider are still present in the VM's VMDK descriptor file

 

Attempting to power on the VM fails with an error similar to the below.  

From the /var/run/log/hostd.log file: 

Power on failed: (vim.fault.GenericVmConfigFault) {
   faultCause = (vmodl.MethodFault) null,
   faultMessage = (vmodl.LocalizableMessage) [
      (vmodl.LocalizableMessage) {
         key = "msg.iofilter.pluginError",
         arg = <unset>,
         message = "Plugin loading error"
      },
      (vmodl.LocalizableMessage) {
         key = "msg.disk.noBackEnd",
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               key = "1",
               value = "/vmfs/volumes/vsan:################-################/########-####-####-####-############/VMNAME_1.vmdk"
            }
         ],
         message = "Cannot open the disk '/vmfs/volumes/vsan:################-################/########-####-####-####-############/VMNAME_1.vmdk' or one of the snapshot disks it depends on. "
      },
      (vmodl.LocalizableMessage) {
         key = "msg.moduletable.powerOnFailed",
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               key = "1",
               value = "Disk"
            }
         ],
         message = "Module 'Disk' power on failed. "
      },
      (vmodl.LocalizableMessage) {
         key = "msg.iofilter.pluginError",
         arg = <unset>,
         message = "Plugin loading error"
      },
      (vmodl.LocalizableMessage) {
         key = "msg.disk.noBackEnd",
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               key = "1",
               value = "/vmfs/volumes/vsan:################-################/########-####-####-####-############/VMNAME.vmdk"
            }
         ],
         message = "Cannot open the disk '/vmfs/volumes/vsan:################-################/########-####-####-####-############/VMNAME.vmdk' or one of the snapshot disks it depends on. "
      },
      (vmodl.LocalizableMessage) {
         key = "msg.vmx.poweron.failed",
         arg = <unset>,
         message = "Failed to start the virtual machine."
      }
   ],
   reason = "Plugin loading error"
   msg = "Plugin loading error"
}

 

When we review the vmdk descriptor files for the vmdk's in the error above, they contain references to the removed IO Filter provider.  .

db.adapterType = "lslogic"
db.deletable = "true"
db.geometry.cylinders = "294084"
db.geometry.heds = "255"
db.geometry.sectors = "63"
db.iofilters = "emcsplitter"   <--- HERE
db.logContentID = "208e80f99f756c948df2da1c1ca6b10"
db.sidecars = "emcsplitter_1, VMNAME_1-58e15e6a5fbefd15.vmdk"    <--- HERE
db.thinProvisioned = "1"
db.toolsInstallTpe = "1"
db.toolsVersion = "12389"
db.uuid = ## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ## "
db.vertualHWVersion = "4"

 

Resolution

Using "vi", we removed these line entries from both vmdk descriptor files.  

After Edit: 

db.adapterType = "lslogic"
db.deletable = "true"
db.geometry.cylinders = "294084"
db.geometry.heds = "255"
db.geometry.sectors = "63"
db.logContentID = "208e80f99f756c948df2da1c1ca6b10"
db.thinProvisioned = "1"
db.toolsInstallTpe = "1"
db.toolsVersion = "12389"
db.uuid = "## ## ## ## ## ## ## ##-## ## ## ## ## ## ## ##"
db.vertualHWVersion = "4"

Note:  DO NOT edit the descriptor without having a backup of it first.  Copy the descriptor file to a safe location prior to editing.

 

Once edited, reload the VM to inventory using ESXi CLI (see KB 343248 for detailed steps).

vim-cmd vmsvc/reload VMid

 

Power on the VM from ESXi CLI (see KB 345049 for detailed steps)

vim-cmd vmsvc/power.on VMid

 

In the vCenter UI, confirm that all actions against the VM are working (power on-off / vMotion, etc.).

Additional Information

Reloading a vmx file without removing the virtual machine from inventory
https://knowledge.broadcom.com/external/article/343248/reloading-a-vmx-file-without-removing-th.html

Powering on a virtual machine from the command line when the host cannot be managed using VMware vSphere Client
https://knowledge.broadcom.com/external/article/345049/powering-on-a-virtual-machine-from-the-c.html