Virtual machines inaccessible with "State in Doubt" and "VMFS Heartbeat Timeouts" reported in logs
search cancel

Virtual machines inaccessible with "State in Doubt" and "VMFS Heartbeat Timeouts" reported in logs

book

Article ID: 432799

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Guest application may experience performance delays, leading to crashes or "Not Responding" states.

  • Virtual machines may freeze , preventing any administrative or user access via Remote Desktop (RDP), SSH, or the console.

  • The vSphere Client may report virtual machines as Disconnected, Orphaned, or otherwise unreachable.

  • The ESXi host loses communication with volume metadata, triggering heartbeat failures that indicate an inability to maintain locks on the datastore.

Environment

VMware vSphere ESXi 8.x
VMware vSphere ESXi 9.x

Cause

The issue is caused by instability within the external storage or SAN fabric layer, leading the ESXi host to report a failure in maintaining a consistent I/O path to the storage devices.
The "State in doubt" message indicates that the storage stack is unable to determine the status of the device, which is typically a result of physical layer interruptions or congestion at the hardware level.

From ESXi, var/run/log/vmkernel.log  shows PDL errors triggering path failovers.

2026-04-26T02:26:20.067Z cpu41:2098231)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x84 (0x45b99727ea48, 0) to dev "naa.#########################" on path "vmhba3:C0:T7:L47" Failed:
2026-04-26T02:26:20.067Z cpu41:2098231)NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0. Act:FAILOVER. cmdId.initiator=0x43091f8cb5e0 CmdSN 0x3ebb0e8
2026-04-26T02:26:20.067Z cpu41:2098231)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.#########################"": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
2026-04-26T02:26:20.551Z cpu16:2097704)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:173: Could not find relative target port ID for path "vmhba3:C0:T7:L47" - Not found (195887107)
2026-04-26T02:26:20.551Z cpu58:2098149)WARNING: NMP: nmpDeviceAttemptFailover:647: Retry world failover device "naa.#########################"" - issuing command 0x45b99727ea48
2026-04-26T02:26:20.552Z cpu7:2098231)WARNING: NMP: nmpCompleteRetryForPath:364: Retry cmd 0x84 (0x45b99727ea48) to dev "naa.#########################"" failed on path "vmhba3:C0:T7:L47" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.
2026-04-26T02:26:20.552Z cpu7:2098231)WARNING: NMP: nmp_PathDetermineFailure:3527: Cmd (0x84) PDL error (0x5/0x25/0x0) - path vmhba3:C0:T7:L47 device naa.#########################" - triggering path failover.

Resolution

Engage the Storage  and Fabric vendors to investigate the health of the storage array and the physical paths (switches, cables, and SFPs).

 

Additional Information

Powered by Wolken Software