Multiple Virtual Machines are Inaccessible Due to VMFS Datastore Extent Offline (Space Exhausted)
search cancel

Multiple Virtual Machines are Inaccessible Due to VMFS Datastore Extent Offline (Space Exhausted)

book

Article ID: 433984

calendar_today

Updated On:

Products

VMware vCenter Server

Issue/Introduction

Symptoms:

  • Users may report that multiple virtual machines are unresponsive, disconnected, or showing as "Inaccessible" in vSphere.
  • The following symptoms are observed:

    • Affected Datastores show 0 bytes capacity and usage.

    • The datastores are configured using multiple extents (spanned volumes).

    • VMs residing on the datastores fail to perform any I/O operations.

    • Attempts to browse the datastore may result in errors or empty folder views.

  • Storage Quota Exceeded: The vobd.log shows the initial warning.
  • Extent Offline: Subsequent entries confirm the device is no longer reachable by the VMFS driver.
  • SCSI Sense Codes: The vmkernel.log reveals the specific hardware error:

 

  • The following lines in /var/log/vobd.log are an "Early Warning" system. Storage (LUN17 and LUN18) is set up to grow as you add data, but it has now reached a pre-set limit (threshold).

vobd[2098148]: [scsiCorrelator] 8084202180921us: [esx.problem.scsi.device.thinprov.atquota] Space utilization on thin-provisioned device naa.xxxxxxxxxxxxxxxxxxxxxxx exceeded configured threshold. Affected datastores (if any): "LUN17".
vobd[2098148]: [scsiCorrelator] 8084211632951us: [esx.problem.scsi.device.thinprov.atquota] Space utilization on thin-provisioned device naa.xxxxxxxxxxxxxxxxxxxxxxx exceeded configured threshold. Affected datastores (if any): "LUN18".

  • These lines in /var/log/vobd.log  indicate a "Critical Disconnect." The system can no longer see or talk to the storage drives (LUN17 and LUN18). It’s the equivalent of someone unplugging an external hard drive while you were using it.

vobd[2098147]: [vmfsCorrelator] 3276012549563us: [vob.vmfs.extent.offline] An attached device went offline. naa.xxxxxxxxxxxxxxxxxxxxxxx:1 file system [LUN17,69afa266-xxxxxxxxxxxxxxxxxxxxxxxx]
vobd[2098148]: [vmfsCorrelator] 8280106594829us: [vob.vmfs.extent.offline] An attached device went offline. naa.xxxxxxxxxxxxxxxxxxxxxxx:1 file system LUN18, 69b147a1-xxxxxxxxxxxxxxxxxxxxxxxx]

  • The system tries to look at the "Table of Contents" for the storage drive to see how it’s organized, but it couldn't read it. Because it can't read this map, it doesn't know where any of the data is stored. 
  • The following lines are seen in /var/log/vmkernel.log

vmkernel: cpu50:8882464)Partition: 1205: Failed to read protective mbr on "naa.xxxxxxxxxxxxxxxxxxxxxxx" : I/O error
vmkwarning: cpu50:8882464)WARNING: Partition: 1387: Partition table read from device naa.xxxxxxxxxxxxxxxxxxxxxxx failed: I/O error
vmkernel: cpu1:2098929)NMP: nmp_ThrottleLogForDevice:3825: last error status from device naa.xxxxxxxxxxxxxxxxxxxxxxx repeated 10 times

  • A specific command (0x8a) to write data to the disk was rejected in /var/log/vmkerne.log. The "sense data" code (0x7 0x27 0x0) specifically means "Write Protected," which indicates the storage array has locked the disk into a "Read-Only" mode because it is completely out of space.

vmkernel: cpu1:2098929)ScsiDeviceIO: 4686: Cmd(0x45bb27bb4fc0) 0x8a, CmdSN 0x34e from world 7797000 to dev "naa.xxxxxxxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x7 0x27 0x0

  • This is a general "Input/Output" failure in /var/log/vmkernel.log. It means the system tried to send or receive data from the drive, but the communication failed completely.

vmkernel: cpu50:8882464)Partition: 477: Failed read for "naa.xxxxxxxxxxxxxxxxxxxxxxx": I/O error
vmkernel: cpu50:8882464)Partition: 1205: Failed to read protective mbr on "naa.xxxxxxxxxxxxxxxxxxxxxxx" : I/O error
vmkwarning: cpu50:8882464)WARNING: Partition: 1387: Partition table read from device naa.xxxxxxxxxxxxxxxxxxxxxxx failed: I/O error

Environment

VMware vSphere ESXi 8.x

Cause

  • The issue is caused by physical capacity exhaustion on the underlying thin-provisioned storage array.
  • When a thin-provisioned LUN reaches its hard limit or a defined quota at the array level, it triggers a "Space Exhausted" condition.
  • The datastores are spanned across multiple extents, the failure of even one physical LUN (device) causes the entire VMFS volume to go offline or enter a degraded, read-only state to prevent data corruption.

Resolution

1. Address Backend Storage Exhaustion

Contact your storage administration team or vendor to:

  • Identify the specific Storage Pool or LUNs associated with the reported naa IDs.
  • Increase the physical capacity of the thin-provisioned pool or expand the volume quota.
  • Ensure the storage array has cleared the "Write Protect" or "Block" status on the LUNs.

2. Rescan Storage on ESXi Hosts

Once the storage array has available space, the ESXi hosts must be notified of the change. Run the following command via SSH on all affected hosts:

esxcli storage core adapter rescan --all

3. Refresh VMFS Volumes

  • Force the VMFS driver to refresh the volume metadata and re-mount the missing extents: vmkfstools -V

4. Verification

  • Check the Datastore view in vSphere to ensure capacity and usage are reporting correctly.
  • Verify that all extents are visible by running: esxcli storage vmfs extent list
  • Once the datastore status is "Normal," right-click the inaccessible VMs and select Refresh Storage Info or unregister/re-register them if they do not automatically recover.
Powered by Wolken Software