VMFS extent Offline, causing VMs and Hostd go unresponsive

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

When a multi-extent VMFS volume loses access to one or more of its backing LUNs, the following behaviors may occur:

Virtual Disk files (= VMDK) residing on the affected Datastore/Disk: I/O might slow down or fail. This might cause Virtual Machines that have VMDKs residing on that device to become unresponsive or even fail.
vSphere Client: The datastore displays 0B capacity and 0B free space.
Datastore capacity mismatch between vCenter/ESXi and actual device capacity from storage array end.
Host Unresponsiveness: The hostd process may become unresponsive, causing the ESXi host to show as "not responding."

This problem does not occur when a non-head extent of the spanned VMFS datastore fails along with the head extent. In this case, the entire Datastore becomes inaccessible and no longer allows I/Os.
In contrast, when only a non-head extent fails, but the head extent remains accessible, the Datastore heartbeat appears to be normal. And the I/Os between the Host and the Datastore continue. However, any I/Os that depend on the failed non-head extent start failing as well. Other I/O transactions might accumulate while waiting for the failing I/Os to resolve and cause the Host to enter the non-responding state ( Reference ) .
If the affected VMFS Datastore capacity was expanded by adding an extent and the extent is offline, VMFS expansion will fail.
Validate if the LUN used for the extent is in detached list

esxcli storage core device detached list

Storage Adapter Rescan fails with the error "An error occurred while communicating with remote host"
var/run/log/vmkwarning.log shows the following:

YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu##:###### opID=#####)WARNING: LVM: 17711: An attached device went offline. ##############:1 file system [VOLUME-NAME, VOLUME-UUID]

/var/run/log/vobd.log shows the following:

YYYY-MM-DDTHH:MM.SSSZ In(14) vobd[2098027]: [vmfsCorrelator] 13900876943838us: [vob.vmfs.extent.offline] An attached device went offline. ##############:1 file system [VOLUME-NAME, VOLUME-UUID]
YYYY-MM-DDTHH:MM.SSSZ In(14) vobd[2098027]: [vmfsCorrelator] 13900800936747us: [esx.problem.vmfs.extent.offline] An attached device ##############:1 may be offline. The file system [VOLUME-NAME, VOLUME-UUID] is now in a degraded state. While the datastore is still available, parts of data that reside on the extent that went offline might be inaccessible.
Running the command "vmkfstools -Ph -v10 /vmfs/volumes/<Datastore_Name>" on the affected host confirms that the datastore is spanned across multiple LUNs and one of the LUN is currently offline.

vmkfstools -Ph -v10 /vmfs/volumes/<Datastore_Name>
VMFS-6.82 (Raw Major Version: 24) file system spanning 2 partitions.
Logical device: ########-########-####-############
Partitions spanned (on "lvm"):
naa.################################:1
(device naa.################################:1 might be offline)
(One or more partitions spanned by this volume may be offline)

The /var/run/log/vmkernel.log show messages referring to "Address temporarily unmapped".

Examples:

YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu47:2812483)Fil6: 4289: 'DATASTORE-NAME': Fil6 file IO (<FD c52 r1>) : Address temporarily unmapped
YYYY-MM-DDTHH:MM.SSSZ Db(167) Hostd[2099105]: --> Failed to copy source (/vmfs/volumes/##############/##########/########.vmdk) to destination (/vmfs/volumes/##############/##########/########.vmdk): Address temporarily unmapped.
YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu21:3221033)WARNING: SVM: 2891: scsi##### Failed SVMFDSIoctlMoveData: Address temporarily unmapped

Environment

ESXi 8.x
ESXi 7.x

Cause

This issue occurs when a multi-extent VMFS volume loses connectivity to one or more backing LUNs, often due to SAN unmapping, cable failure, or LUN reformatting.
The /var/run/log/vmkernel.log file confirms that the device was either removed or became permanently inaccessible.

YYYY-MM-DDTHH:MM.SSSZ Wa(180) vmkwarning: cpu80:2097830)WARNING: ScsiDevice: 1794: Device :naa.################################ has been removed or is permanently inaccessible.
YYYY-MM-DDTHH:MM.SSSZ In(182) vmkernel: cpu80:2097830)ScsiDevice: 1808: Permanently inaccessible device :naa.################################ has no more open connections. It is now safe to unmount datastores (if any) and delete the device.

Resolution

Identify the Storage device(s) backing the affected VMFS Datastore and restore connectivity to these Storage device(s) by following the steps below:

Note:

If the Storage device(s) has been reformatted and reassigned to another Volume, the corresponding portion of the original Volume will be permanently lost and cannot be recovered.

Run the following command to determine which Storage devices back the affected Datastore (= VMFS volume ):
vmkfstools -Ph /vmfs/volumes/<datastore>
Example:
[root@####:~] vmkfstools -Ph /vmfs/volumes/<datastore>
VMFS-6.82 (Raw Major Version: 24) file system spanning 4 partitions.
File system label (if any): datastore
Mode: public
Capacity 399.8 GB, 156.2 GB available, file block size 1 MB, max supported file size 64 TB
Disk Block Size: 512/16384/0
UUID: ########-########-####-############
Partitions spanned (on "lvm"):
#################:1 ----------> First Partition: Head extent.
#################:1
#################:1
#################:1
Is Native Snapshot Capable: NO
Note: In this scenario, the Datastore is configured with multiple Extents. As a result, multiple Storage devices (= ################# ) appear under "Partitions spanned (on 'lvm')" indicating that multiple Devices are backing this Datastore.
Verify whether this is a local Datastore or non-local (= located on SAN Array) by running the following command:
esxcli storage core device list
Example:
#################:
Display Name: Local Make Disk (#################)
Has Settable Display Name: true
Size: 3662830
Device Type: Direct-Access
Multipath Plugin: HPP
Devfs Path: /vmfs/devices/disks/#################
Vendor: "Vendor Name"
Model: ######
Revision: ######
SCSI Level: 6
Is Pseudo: false
Status: on
Is RDM Capable: true
Is Local: true

Look in the output for the Storage device(s) ################# listed under "Partitions spanned (on "lvm")"

Check for "Is Local:" from the above output:
Is Local: true -------> This is a local Storage device (= affected Datastore is local to the Host)
Is Local: false -------> This is a Storage device which is not local to the Host. The Storage device is located on SAN (= SAN LUN / external to the Host). The affected Datastore is located on SAN.
If one or more of the Storage device(s) ################# listed under "Partitions spanned (on "lvm")" do show Is Local: true, identify their physical location:

esxcli storage core device physical get -d #################
Physical Location: enclosure ### slot ###
Check the Health of the Storage device(s) ################# listed under "Partitions spanned (on "lvm")"
If the device is in detached state then attach it and see if that helps.
Involve your HW Storage Vendor to identify and fix any Storage device issues or have them check the Health of the Storage device(s)
If the issue continues after resolving the underlying Storage problem, restart the affected ESXi Host(s)