VM power on fails with error "File system specific implementation of GetPageRef[file] failed"

Products

• VMware vSphere ESXi 7.0
• VMware vSphere ESXi 8.0

Issue/Introduction

• VM power-on fails with the error "File system specific implementation of GetPageRef[file] failed"
• In vSphere UI we see below the error messages :

Object not found Failed to start the virtual machine. Cannot open the disk '/vmfs/volumes/#####-59862dec-#####-########/###############/###############_11.vmdk' or one of the snapshot disks it depends on.

File system specific implementation of GetPageRef[file] failed File system specific implementation of GetPageRef[file] failed Cannot open the disk '/vmfs/volumes/#####-59862dec-#####-########/###############/###############.vmdk' or one of the snapshot disks it depends on.
Cannot open the disk '/vmfs/volumes/#####-59862dec-#####-########/###############/###############_15.vmdk' or one of the snapshot disks it depends on. Module Disk power on failed.
Cannot open the disk '/vmfs/volumes/#####-59862dec-#####-########/###############/###############_25.vmdk' or one of the snapshot disks it depends on.

• Powering on via ESXi SSH reports below error :
  
  [root@########:/vmfs/volumes/62988c36-59862dec-2ba1-########/########] vim-cmd vmsvc/power.on 3
Powering on VM:
Power on failed: (vim.fault.GenericVmConfigFault) {
   faultCause = (vmodl.MethodFault) null,
   faultMessage = (vmodl.LocalizableMessage) [
      (vmodl.LocalizableMessage) {
         key = "msg.iofilter.notFound",
         arg = <unset>,
         message = "Object not found"
      },
      (vmodl.LocalizableMessage) {
         key = "msg.disk.noBackEnd",
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               key = "1",
               value = "/vmfs/volumes/62988c36-59862dec-2ba1-########/########/########_15.vmdk"
            }
         ],
         message = "Cannot open the disk '/vmfs/volumes/62988c36-59862dec-2ba1-########/########/########_15.vmdk' or one of the snapshot disks it depends on. "
      },
      (vmodl.LocalizableMessage) {
         key = "msg.moduletable.powerOnFailed",
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               key = "1",
               value = "Disk"
            }
         ],
         message = "Module 'Disk' power on failed. "
      },
      (vmodl.LocalizableMessage) {
         key = "vob.fssvec.GetPageRef.file.failed",
         arg = <unset>,
         message = "File system specific implementation of GetPageRef[file] failed"
      },
      (vmodl.LocalizableMessage) {
         key = "vob.fssvec.GetPageRef.file.failed",
         arg = <unset>,
         message = "File system specific implementation of GetPageRef[file] failed"
      },
      (vmodl.LocalizableMessage) {
         key = "msg.iofilter.notFound",
         arg = <unset>,
         message = "Object not found"
      },
      },
      (vmodl.LocalizableMessage) {
         key = "msg.disk.noBackEnd",
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               key = "1",
               value = "/vmfs/volumes/62988c36-59862dec-2ba1-########/########/########.vmdk"
            }
         ],
         message = "Cannot open the disk '/vmfs/volumes/62988c36-59862dec-2ba1-########/########/########.vmdk' or one of the snapshot disks it depends on. "
      },
      (vmodl.LocalizableMessage) {
         key = "msg.iofilter.notFound",
         arg = <unset>,
         message = "Object not found"
      },
      (vmodl.LocalizableMessage) {
         key = "msg.disk.noBackEnd",
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               key = "1",
               value = "/vmfs/volumes/62988c36-59862dec-2ba1-########/########/########_11.vmdk"
            }
         ],
         message = "Cannot open the disk '/vmfs/volumes/62988c36-59862dec-2ba1-########/########/########_11.vmdk' or one of the snapshot disks it depends on. "
      },
      (vmodl.LocalizableMessage) {
         key = "msg.iofilter.notFound",
         arg = <unset>,
         message = "Object not found"
      },
      (vmodl.LocalizableMessage) {
         key = "msg.disk.noBackEnd",
         arg = (vmodl.KeyAnyValue) [
            (vmodl.KeyAnyValue) {
               key = "1",
               value = "/vmfs/volumes/62988c36-59862dec-2ba1-########/########/########_25.vmdk"
            }
         ],
         message = "Cannot open the disk '/vmfs/volumes/62988c36-59862dec-2ba1-########/########/########_25.vmdk' or one of the snapshot disks it depends on. "
      },
      (vmodl.LocalizableMessage) {
         key = "msg.vmx.poweron.failed",
         arg = <unset>,
         message = "Failed to start the virtual machine."
      }
   ],
   reason = "Object not found"
   msg = "Object not found"
  
  
  
  
  

Environment

• VMware ESXi 7.x
• VMware ESXi 8.x
• Dell EMC PowerProtect

Cause

The Virtual Machine was protected by Dell EMC PowerProtect and this created vmfd files (files that contain metadata for vmdk disks). The virtual disks that were complaining about "msg.disk.noBackEnd" had associated vmfd files missing. 

Resolution

Restored the missing vmdk from the backup and then attached the restored disk to the virtual machine.