ESXi Host slow/unresponsive due to storage issues resulting in VMs going unresponsive.
search cancel

ESXi Host slow/unresponsive due to storage issues resulting in VMs going unresponsive.

book

Article ID: 392873

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Operations to the host via vCenter Server are significantly delayed or time out.

  • Browsing datastores takes an excessive amount of time.

  • Virtual Machines (VMs) appear powered on but are unresponsive to console or network access.

  • vSphere HA may trigger a reset action for affected VMs.

Environment

VMware vSphere ESXi (All Versions)

Cause

This is cause due to one or more datastores having poor performance resulting in heartbeats failing and eventually hostd crashes resulting in the host becoming completely unresponsive. You'll see var/run/log/vmkernel.log flooded with similar messages:

YYYY-MM-DDTHH:MM:SS.567Z cpu16:2097877)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8645 microseconds to 195723 microseconds.
YYYY-MM-DDTHH:MM:SS.766Z cpu16:2097877)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8645 microseconds to 392920 microseconds.
YYYY-MM-DDTHH:MM:SS.195Z cpu11:2097872)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8645 microseconds to 192213 microseconds.
.

YYYY-MM-DDTHH:MM:SS.966Z cpu0:2097227)ScsiDeviceIO: 4087: Cmd(0x45b8e4830848) 0x2a, cmdId.initiator=0x430a81c45900 CmdSN 0x8000007f from world 2105834 to dev "naa.6000d31######################03" failed H:0x8 D:0x0 P:0x0 Cancelled from device
YYYY-MM-DDTHH:MM:SS.869Z cpu31:2097483)ScsiDeviceIO: 4154: Cmd(0x45d8d6009d08) 0x28, cmdId.initiator=0x43062f340c80 CmdSN 0x80aec4f from world 2613488 to dev "naa.6000d31######################03" failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer

YYYY-MM-DDTHH:MM:SS.843Z cpu17:2098132)HBX: 294: 'SAS Datastore': HB at offset 3899392 - Reclaimed heartbeat [Timeout]:
YYYY-MM-DDTHH:MM:SS.843Z cpu17:2098132)  [HB state abcdef02 offset 3899392 gen 863 stampUS 831516988653 uuid 67bf65e0-########-326a-############ jrnl <FB 12> drv 24.82 lockImpl 4 ip 10.#.##.##]

YYYY-MM-DDTHH:MM:SS.707Z cpu36:2097483)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d31######################03" state in doubt; requested fast path state update...

Resolution

 

  1. Reboot the Host: Perform a reboot of the affected ESXi host to clear transient I/O contention and restore management service responsiveness.

  2. Mitigate Storage Congestion: Coordinate with your storage administration team to investigate the SAN fabric and storage array for performance bottlenecks or hardware faults.

  3. Clear Stale Locks: If VMs remain hung after storage latency is resolved, perform a forced lock release on the specific LUN or volume.

  4. Hardware Health Check: Ensure SAN switch firmware, HBA drivers, and server BIOS are up to date and match the VMware Compatibility Guide.

Additional Information

Powered by Wolken Software