Unable to power on a VM that resides on a multi extent VMFS datastore.

Products

• VMware vSphere ESXi
• VMware vSphere ESX 8.x
• VMware vSphere ESX 7.x

Issue/Introduction

When powering on a virtual machine that resides on a multi-extent datastore, the task fails with the following error in the vSphere UI:

Power On this virtual machine
StateFailed - File system specific implementation of GetPageRef[file] failed
Errors
File system specific implementation of GetPageRef[file] failedFile system specific implementation of FileIO[file] failed14 (Bad address)Cannot open the disk '/vmfs/volumes/<Datastore-UUID>2/<VM-Name>/<VM-Name>-000001.vmdk' or one of the snapshot disks it depends on.

 

Environment

VMware vSphere ESXi  

Cause

When a device from a multi-extent datastore is administratively turned off, i.e. detached in the vSphere UI, ESXi will detect that device as offline. 

 vmkfstools -Ph -v10 /vmfs/volumes/<Datastore-Name>
VMFS-6.82 (Raw Major Version: 24) file system spanning 2 partitions.
File system label (if any): Datastore-Name
Mode: public
Capacity 52.8 TB, 1.1 TB available, file block size 1 MB, max supported file size 64 TB
Volume Creation Time: Wed Mar ## ##:##:## ####
...
Partitions spanned (on "lvm"):
        naa.###############################:1
        (device naa.###############################:1 might be offline)
        (One or more partitions spanned by this volume may be offline)
... 

If the device is not the primary extent, the datastore will remain online, but any VM operation which requires accessing the offline extent will fail. 

 

If a device backing a VMFS file system has been detached /var/log/vobd.log will report: 

####-##-##T##:##:##.###Z: [vmfsCorrelator] 450164542us: [esx.problem.vmfs.extent.offline] An attached device naa.###############################:1 may be offline. The file system [Datastore-Name, ########-########-####-############] is now in a degraded state. While the datastore is still available, parts of data that reside on the extent that went offline might be inaccessible.
 

Detached device status can be confirmed from vSphere UI or CLII:

In the cli, run this command:
esxcli storage core device detached list

 

Resolution

In the vSphere Client, select a host in Hosts and Clusters view.
Then select: Configure - Storage devices - Device (e.g. naa.#########################) - Attach

Confirm the the operational state of the device is attached on all hosts in the cluster.