Virtual machines have become unresponsive or slow resulting in VSCSI resets
Article ID: 392870
Updated On:
Products
Issue/Introduction
Virtual Machine performance is sluggish or it becomes unresponsive.
Following logs snippets are observed at /var/run/log/vmkernel.log on the ESXi host.
YYYY-MM-DDTHH:MM:SS.653Z cpu57:8916482)VSCSI: 2973: handle 38295998585421404(GID:xxxxx)(vscsi0:0):Added handle (refCnt = 3) to vscsiResetHandleList vscsiResetHandleCount = 1YYYY-MM-DDTHH:MM:SS.653Z cpu14:2097732)VSCSI: 3226: handle 38295998585421404(GID:xxxxx)(vscsi0:0):processing reset for handle ... state 1381192707YYYY-MM-DDTHH:MM:SS.653Z cpu14:2097732)VSCSI: 3335: handle 38295998585421404(GID:xxxxx)(vscsi0:0):Reset [Retries: 0/0] from (vmm0:TESTVM)YYYY-MM-DDTHH:MM:SS.157Z cpu14:2097732)VSCSI: 3226: handle 38295998585421404(GID:xxxxx)(vscsi0:0):processing reset for handle ... state 1381192706YYYY-MM-DDTHH:MM:SS.659Z cpu14:2097732)VSCSI: 3226: handle 38295998585421404(GID:xxxxx)(vscsi0:0):processing reset for handle ... state 1381192706YYYY-MM-DDTHH:MM:SS.655Z cpu57:8916482)WARNING: VSCSI: 3967: handle 38295998585421404(GID:xxxxx)(vscsi0:0):WaitForCIF: Issuing reset; number of CIF:4YYYY-MM-DDTHH:MM:SS.655Z cpu57:8916482)WARNING: VSCSI: 2986: handle 38295998585421404(GID:xxxxx)(vscsi0:0):Ignoring double resetYYYY-MM-DDTHH:MM:SS.864Z cpu3:2097732)VSCSI: 3226: handle 38295998585421404(GID:xxxxx)(vscsi0:0):processing reset for handle ... state 1381192706YYYY-MM-DDTHH:MM:SS.367Z cpu3:2097732)VSCSI: 3226: handle 38295998585421404(GID:xxxxx)(vscsi0:0):processing reset for handle ... state 1381192706YYYY-MM-DDTHH:MM:SS.840Z cpu3:2097732)VSCSI: 3226: handle 38295998585421404(GID:xxxxx)(vscsi0:0):processing reset for handle ... state 1381192706YYYY-MM-DDTHH:MM:SS.840Z cpu3:2097732)VSCSI: 3335: handle 38295998585421404(GID:xxxxx)(vscsi0:0):Reset [Retries: 15/0] from (vmm0:TESTVM)YYYY-MM-DDTHH:MM:SS.343Z cpu3:2097732)VSCSI: 3226: handle 38295998585421404(GID:xxxxx)(vscsi0:0):processing reset for handle ... state 1381192706
Environment
VMware vSphere ESXi (All Versions)
Cause
The datastore on which the VMs reside is experiencing performance degradation, including aborts and resets. The following log snippets are observed at /var/run/log/vmkernel.log and /var/run/log/vmkwarning.log on the ESXi host.
YYYY-MM-DDTHH:MM:SS.653Z cpu16:2097877)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8645 microseconds to 195723 microseconds.YYYY-MM-DDTHH:MM:SS.653Z cpu16:2097877)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8645 microseconds to 392920 microseconds.YYYY-MM-DDTHH:MM:SS.653Z cpu11:2097872)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8645 microseconds to 192213 microseconds.YYYY-MM-DDTHH:MM:SS.653Z cpu8:2097869)WARNING: ScsiDeviceIO: 1513: Device naa.6000d31######################03 performance has deteriorated. I/O latency increased from average value of 8649 microseconds to 215719 microseconds.
YYYY-MM-DDTHH:MM:SS.653Z cpu0:2097227)ScsiDeviceIO: 4087: Cmd(0x45b8e4830848) 0x2a, cmdId.initiator=0x430a81c45900 CmdSN 0x8000007f from world 2105834 to dev "naa.6000d31######################03" failed H:0x8 D:0x0 P:0x0 Cancelled from deviceYYYY-MM-DDTHH:MM:SS.653Z cpu31:2097483)ScsiDeviceIO: 4154: Cmd(0x45d8d6009d08) 0x28, cmdId.initiator=0x43062f340c80 CmdSN 0x80aec4f from world 2613488 to dev "naa.6000d31######################03" failed H:0x5 D:0x0 P:0x0 Cancelled from driver layer
Resolution
This issue appears to stem from an underlying storage problem. As a temporary measure, a host reboot can be attempted to ascertain if the issue is resolved. It is recommended to engage the storage vendor to diagnose any underlying issues with the storage infrastructure.
Additional Information
Feedback
