ESXi Host slow/unresponsive due to storage issues resulting in VMs going unresponsive.
Article ID: 392873
Updated On:
Products
Issue/Introduction
- Operations to the host using vCenter are slow.
- Trying to browse data in the datastores is taking too long.
- Cannot access some VMs, the VMs are powered on but are unable to open remote console.
- Host eventually becomes unresponsive
Environment
VMware vSphere ESXi (All Versions)
Cause
This is cause due to one or more datastores having poor performance resulting in heartbeats failing and eventually hostd crashes resulting in the host becoming completely unresponsive. You'll see vmkernel flooded with similar messages:
2025-03-08T09:44:13.567Z cpu16:2097877)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8645 microseconds to 195723 microseconds.
2025-03-08T09:44:14.766Z cpu16:2097877)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8645 microseconds to 392920 microseconds.
2025-03-08T09:44:18.195Z cpu11:2097872)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8645 microseconds to 192213 microseconds.
.
2025-03-08T10:03:16.966Z cpu0:2097227)ScsiDeviceIO: 4087: Cmd(0x45b8e4830848) 0x2a, cmdId.initiator=0x430a81c45900 CmdSN 0x8000007f from world 2105834 to dev "naa.6000d31######################03" failed H:0x8 D:0x0 P:0x0 Cancelled from device
2025-03-08T19:11:50.869Z cpu31:2097483)ScsiDeviceIO: 4154: Cmd(0x45d8d6009d08) 0x28, cmdId.initiator=0x43062f340c80 CmdSN 0x80aec4f from world 2613488 to dev "naa.6000d31######################03" failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer
2025-03-08T10:03:29.843Z cpu17:2098132)HBX: 294: 'SAS Datastore': HB at offset 3899392 - Reclaimed heartbeat [Timeout]:
2025-03-08T10:03:29.843Z cpu17:2098132) [HB state abcdef02 offset 3899392 gen 863 stampUS 831516988653 uuid 67bf65e0-########-326a-############ jrnl <FB 12> drv 24.82 lockImpl 4 ip 10.#.##.##]
2025-03-08T20:52:11.707Z cpu36:2097483)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d31######################03" state in doubt; requested fast path state update...
Resolution
Reboot the host to restore functionality and then have the poorly performing datastores investigated by storage vendor/team
Additional Information
Feedback
