Virtual machines periodically freeze/hang due to slow storage response to write-same

Products

• VMware vSphere ESXi
• VMware vSphere ESX 7.x
• VMware vSphere ESX 8.x

Issue/Introduction

• VMs intermittently freeze/hang during which they lose network connectivity for a short amount of time.
• This occurs randomly, e.g. a few times a day with the VM becoming unresponsive for a few seconds or minutes.
• When a VM becomes unresponsive the windows event viewer reports the event: "Warning,DD/MM/YYYY hh:mm:ss,LSI_SAS,129,None,"Reset to device, \Device\RaidPort0, was issued."

 

 

Environment

VMware vSphere ESXi

Cause

VMs periodically freeze/hang due to slow storage response to write-same commands (Cmd 0x93).

In the /var/log/vmkernel.log at time the VM becomes hung the ESX will report continuous vscsi resets, this is then followed by aborts for a Write-Same (cmd 0x93)

ESXi: /var/log/vmkernel.log:
####-##-##T##:##:##.###Z In(182) vmkernel: cpu1:2097634)VSCSI: 3738: handle ##################(GID:#####)(vscsi0:#):processing reset for handle ... state 1381192706
####-##-##T##:##:##.###Z In(182) vmkernel: cpu1:2097634)VSCSI: 3738: handle ##################(GID:#####)(vscsi0:#):processing reset for handle ... state 1381192706
####-##-##T##:##:##.###Z In(182) vmkernel: cpu1:2097634)VSCSI: 3738: handle ##################(GID:#####)(vscsi0:#):processing reset for handle ... state 1381192706
####-##-##T##:##:##.###Z In(182) vmkernel: cpu1:2097634)VSCSI: 3738: handle ##################(GID:#####)(vscsi0:#):processing reset for handle ... state 1381192706
####-##-##T##:##:##.###Z In(182) vmkernel: cpu18:2097672)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0x93 (0x45b9537d4780, 48907388) to dev "naa.#################################" on path "vmhba64:C0:T0:L2" Failed:
####-##-##T##:##:##.###Z In(182) vmkernel: cpu18:2097672)NMP: nmp_ThrottleLogForDevice:3898: H:0x5 D:0x0 P:0x0 . Act:EVAL. cmdId.initiator=0x4306fb788600 CmdSN 0x83697694
####-##-##T##:##:##.###Z Wa(180) vmkwarning: cpu18:2097672)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:235: NMP device "naa.#################################" state in doubt; requested fast path state update...
####-##-##T##:##:##.###Z In(182) vmkernel: cpu18:2097672)ScsiDeviceIO: 4656: Cmd(0x45b9537d4780) 0x93, cmdId.initiator=0x4306fb788600 CmdSN 0x83697694 from world 48907388 to dev "naa.#################################" failed H:0x5 D:0x0 P:0x0 Cancelled from driver
####-##-##T##:##:##.###Z In(182) vmkernel: cpu18:2097672)layer. Cmd count Active:1 Queued:0
####-##-##T##:##:##.###Z In(182) vmkernel: cpu1:2097634)VSCSI: 3738: handle ##################(GID:#####)(vscsi0:#):processing reset for handle ... state 1381192706
####-##-##T##:##:##.###Z In(182) vmkernel: cpu1:2097634)VSCSI: 3589: handle ##################(GID:#####)(vscsi0:#):reset processed removed handle from vscsiResetHandleList 0

 

Resolution

Contact your storage vendor to investigate the slow storage array response to write-same commands.