Virtual machines are inaccessible with "State in Doubt" and "VMFS Heartbeat Timeouts" reported in logs
search cancel

Virtual machines are inaccessible with "State in Doubt" and "VMFS Heartbeat Timeouts" reported in logs

book

Article ID: 432799

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Guest application may experience performance delays, leading to crashes or "Not Responding" states.

  • Virtual machines may freeze , preventing any administrative or user access via Remote Desktop (RDP), SSH, or the console.

  • The vSphere Client may report virtual machines as Disconnected, Orphaned, or otherwise unreachable.

  • The ESXi host loses communication with the volume metadata, triggering heartbeat failures that indicate an inability to maintain locks on the datastore.

Environment

  • VMware vSphere ESXi 8.x
  • VMware vSphere ESXi 9.x

Cause

  • The issue is caused by instability within the external storage or SAN fabric layer, leading the ESXi host to report a failure in maintaining a consistent I/O path to the storage devices.

  • The "State in doubt" message indicates that the storage stack is unable to determine the status of the device, which is typically a result of physical layer interruptions or congestion at the hardware level.

  • The ESXi host var/run/log/vmkernel.log logs show PDL errors triggering path failovers.

    YYYY-MM-DDTHH:MM:SS.MS+HH:MM cpu##:2098231)NMP: nmp_ThrottleLogForDevice:3867: Cmd 0x84 (0x45b####7ea48, 0) to dev "naa.#########################" on path "vmhba##:C0:T7:L##" Failed:
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM cpu##:2098231)NMP: nmp_ThrottleLogForDevice:3875: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0. Act:FAILOVER. cmdId.initiator=0x43####b5e0 CmdSN 0x####0e8
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM cpu##:2098231)WARNING: NMP: nmp_DeviceRetryCommand:133: Device "naa.#########################"": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM cpu##:2097704)WARNING: VMW_SATP_ALUA: satp_alua_getTargetPortInfo:173: Could not find relative target port ID for path "vmhba##:C0:T7:L##" - Not found (195887107)
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM cpu##:2098149)WARNING: NMP: nmpDeviceAttemptFailover:647: Retry world failover device "naa.#########################"" - issuing command 0x45b####7ea48
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM cpu##:2098231)WARNING: NMP: nmpCompleteRetryForPath:364: Retry cmd 0x84 (0x45b####7ea48) to dev "naa.#########################"" failed on path "vmhba##:C0:T7:L##" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x25 0x0.
    YYYY-MM-DDTHH:MM:SS.MS+HH:MM cpu##:2098231)WARNING: NMP: nmp_PathDetermineFailure:3527: Cmd (0x84) PDL error (0x5/0x25/0x0) - path vmhba##:C0:T7:L## device naa.#########################" - triggering path failover.

Resolution

Engage the Storage and Fabric vendors to investigate the health of the storage array and the physical paths (switches, cables, and SFPs).

Additional Information