Operations to the host via vCenter Server are significantly delayed or time out.
Multiple hosts show not responding intermittently.
Browsing datastores takes an excessive amount of time.
Virtual Machines (VMs) appear powered on but are unresponsive to console or network access or in a disconnected state.
vSphere HA may trigger a reset action for affected VMs.
VMware vSphere ESXi (All Versions)
This is caused due to one or more datastores having poor performance resulting in heartbeats failing and eventually hostd crashes resulting in the host becoming completely unresponsive.
You'll see var/run/log/vmkernel.log flooded with similar messages:
YYYY-MM-DDTHH:MM:SS.567Z cpu16:2097877)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8645 microseconds to 195723 microseconds.YYYY-MM-DDTHH:MM:SS.766Z cpu16:2097877)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8645 microseconds to 392920 microseconds..
YYYY-MM-DDTHH:MM:SS.966Z cpu0:2097227)ScsiDeviceIO: 4087: Cmd(0x45b8e4830848) 0x2a, cmdId.initiator=0x430a81c45900 CmdSN 0x8000007f from world 2105834 to dev "naa.6000d31######################03" failed H:0x8 D:0x0 P:0x0 Cancelled from deviceYYYY-MM-DDTHH:MM:SS.869Z cpu31:2097483)ScsiDeviceIO: 4154: Cmd(0x45d8d6009d08) 0x28, cmdId.initiator=0x43062f340c80 CmdSN 0x80aec4f from world 2613488 to dev "naa.6000d31######################03" failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer
YYYY-MM-DDTHH:MM:SS.843Z cpu17:2098132)HBX: 294: 'Datastore': HB at offset 3899392 - Reclaimed heartbeat [Timeout]:YYYY-MM-DDTHH:MM:SS.843Z cpu17:2098132) [HB state abcdef02 offset 3899392 gen 863 stampUS 831516988653 uuid 67bf65e0-########-326a-############ jrnl <FB 12> drv 24.82 lockImpl 4 ip 10.#.##.##]
YYYY-MM-DDTHH:MM:SS.707Z cpu36:2097483)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237: NMP device "naa.6000d31######################03" state in doubt; requested fast path state update...
Under /var/run/log/hostd.log, there are hostd performance degraded events due to these issues:
YYYY-MM-DDTHH:MM:SS.MSZ In(166) Hostd[2101869]: [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 347521 : Issue detected on cluster. UK in ha-datacenter: hostd performance has degraded due to high system latency