VMFS Datastore went to an inaccessible/unmounted state.

Products

• VMware vSphere ESXi

Issue/Introduction

An administrator wants to understand why a VMFS datastore shows as inaccessible or unmounted, suddenly in the environment, or found and not noticed for a period of time.

Environment

vSphere ESXi (All Versions)

Cause

A volume going into an inaccessible states means the ESXi host can no longer access the volume, regardless of transport type (e.g. SCSI/NFS, etc). 

There are many reasons why a volume may go into an inaccessible state. Such as:

• Write access loss to a volume
• HW issues with network/fabric components
• Presentation/access removed suddenly from the storage array/filer
• Administratively unmounted/detached volumes that was done in an intended way for removal

To understand the cause, you will need to review numerous logs on an ESXi host that exhibited the issue, such as the vobd.log and vmkernel.log. As well, we also need to review the hostd.log as well if we want to ensure it was not removed by an administrator.

All the above logs can be found in /var/run/log on the ESXi host and can be freely accessed from an SSH session.

 

 

Resolution

In the case of the possible cause related to "Administratively unmounted/detached" volumes, we can see in the vobd.log that the volume was unmounted and detached:

20xx-yy-zzT19:08:54.614Z: [vmfsCorrelator] 14463472454682us: [vob.vmfs.volume.umounted] File system '[Datastore_Name, ########-########-####-################]' (LV ########-########-####-################) un-mounted.
20xx-yy-zzT19:08:54.614Z: [vmfsCorrelator] 14463921867045us: [esx.audit.vmfs.volume.umounted] File system [Datastore_Name, ########-########-####-################] on volume ########-########-####-################ has been safely un-mounted. The datastore is no longer accessible on this host.
20xx-yy-zzT19:10:07.731Z: [scsiCorrelator] 14463545569463us: [vob.scsi.device.state.off] Device naa.################################  has been turned off administratively.
2025-02-12T19:10:07.731Z: [scsiCorrelator] 14463994983375us: [esx.problem.scsi.device.state.off] Device: naa.################################  has been turned off administratively.

Further, reviewing hostd.log reveals the user who performed the action:

20xx-yy-zzT19:08:46.504Z info hostd[2100161] [Originator@6876 sub=Vimsvc.TaskManager opID=########-#######-####-####-##:########-##-##-#### user=vpxuser:DOMAIN.COM\domain_user_account] Task Created : haTask-ha-host-vim.host.StorageSystem.unmountVmfsVolume-########

Through this review, in this example, we have concluded that the inaccessible state was intended and done by a member of the virtual or storage administration of the team.