ESXi Host Becomes Unresponsive Due to Storage LUN Data Protect State
search cancel

ESXi Host Becomes Unresponsive Due to Storage LUN Data Protect State

book

Article ID: 451329

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:

  • ESXi host appears as "Not Responding" in vCenter.

  • Unable to access the ESXi host's web client.

  •  Running any esxcli commands fail with error: "Connection failed: 503 Service Unavailable".

  • Even after restarting the ESXi host's management services, the host remains in the "Not Responding" state.

  • VMs running on the host continue to run.

  • df -h command to list the storage hangs and does not complete.

  • ESXi host log bundle generation via SSH would never complete.

  • One or many VMFS datastores may be inaccessible.

Environment

  • VMware vSphere ESXi 7.x

  • VMware vSphere ESXi 8.x

Cause

  • The LUN backing the inaccessible VMFS datastore has entered a DATA PROTECT state, placing the backing VMFS volume into a write-protected state.

  • When a storage array transitions a LUN to a DATA PROTECT state, write operations are rejected by the target storage controller. ESXi continuously attempts to update VMFS heartbeat locks on mounted datastores.

  • The LUN can be confirmed to be in a DATA PROTECT state from the host's /var/run/log/vmkernel.log:

    2026-08-03T04:34:56.191Z In(182) vmkernel: cpu142:2099285)ScsiDeviceIO: 4697: Cmd(0x45dc63f37c00) 0xfe, CmdSN 0x4d1f from world 27521070 to dev "naa.#######" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x7 0x22 0x0

  • hostd tracks and queries volume status synchronously, and since LUN is in DATA PROTECT mode, hostd's internal thread pool becomes completely blocked waiting for I/O operations on the affected LUN's volume to resolve:

    ESXi host's /var/run/log/hostd.log:

    2026-08-03T02:52:52.167Z Wa(164) Hostd[27532874]: [Originator@6876 sub=IoTracker] In thread 27532879, open("/vmfs/volumes/<affected_vmfs_ds_on_LUN>") took over 6 sec.
    .
    .
    2026-08-03T04:34:53.153Z Wa(164) Hostd[27532878]: [Originator@6876 sub=IoTracker] In thread 27532879, open("/vmfs/volumes/<affected_vmfs_ds_on_LUN>") took over 6127 sec.

  • Similarly, due to the ESXi host being unable to write to the LUN, the heartbeat IOs are also failing for the device:

    ESXi host's /var/run/log/vmkernel.log:

    2026-08-03T03:08:51.217Z In(182) vmkernel: cpu48:27502258)HBX: 1019: '<affected_vmfs_ds_on_LUN>': HB at offset ####### - Setting pulse failed: I/O error:
    2026-08-03T03:08:51.217Z In(182) vmkernel: cpu48:27502258)  [HB state abcdef02 offset ####### gen 1 stampUS 511083124#### uuid ########-########-####-############ jrnl <FB 0> drv 14.81 lockImpl 4 ip ###.##.##.###]

  • Running VMs remain operational as they do not use the write protected VMFS datastore.

Resolution

 To resolve this issue:

  1. Ensure storage array access permissions for the affected LUN is restored to Read & Write mode for the ESXI host.

  2. If the LUN is read-only on purpose or no longer required:

    1. Detach LUN from the ESXi host.

    2. Unpresent/unmask the LUN from the storage array.

  3. After resolving storage accessibility or detaching the LUN, restart the ESXi management agents via SSH or direct console: services.sh restart

  4. Verify that the ESXi host reconnects to vCenter Server and that host management functionality (vSphere Client, ESXi Host Client, esxcli) is fully restored.

Additional Information

Similar issue may be observed after failover activity on storage, refer - The datastore cannot be unmounted after performing a Storage DR activity.

Powered by Wolken Software