VDI VM's going into un-responsive state
Article ID: 318792
Updated On:
Products
VMware vSphere ESXi
Issue/Introduction
Symptoms:
vmware.log
=========
2020-12-15T07:45:08.263Z| vcpu-0| I125: Guest: toolbox-dnd: Version: 11.0.5.17716 (build-15389592)
2020-12-15T07:45:08.264Z| vcpu-1| W115: GuestRpc: application toolbox-dnd, changing channel 65535 -> 2
2020-12-15T07:45:08.264Z| vcpu-1| I125: GuestRpc: Channel 2, guest application toolbox-dnd.
2020-12-15T07:45:36.959Z| vmx| I125: GuestRpcSendTimedOut: message to toolbox-dnd timed out.
2020-12-15T07:45:36.959Z| vmx| I125: GuestRpcSendTimedOut: message to toolbox timed out.
2020-12-15T07:45:50.509Z| vcpu-0| I125: Tools: Tools heartbeat timeout.
2020-12-15T07:45:51.959Z| vmx| I125: GuestRpcSendTimedOut: message to toolbox-dnd timed out.
2020-12-15T07:45:51.959Z| vmx| I125: GuestRpc: app toolbox-dnd's second ping timeout; assuming app is down
2020-12-15T07:45:51.959Z| vmx| I125: GuestRpcSendTimedOut: message to toolbox timed out.
2020-12-15T07:45:51.959Z| vmx| I125: GuestRpc: app toolbox's second ping timeout; assuming app is down
2020-12-15T07:45:51.960Z| vmx| I125: GuestRpc: Reinitializing Channel 2(toolbox-dnd)
2020-12-15T07:45:51.960Z| vmx| I125: GuestMsg: Channel 2, Cannot unpost because the previous post is already completed
2020-12-15T07:45:51.960Z| vmx| I125: GuestRpc: Reinitializing Channel 0(toolbox)
2020-12-15T07:45:51.960Z| vmx| I125: GuestMsg: Channel 0, Cannot unpost because the previous post is already completed
2020-12-15T08:59:05.614Z| mks| I125: SSL: syscall error 104: Connection reset by peer
2020-12-15T08:59:05.614Z| mks| I125: SOCKET 4 (133) recv error 104: Connection reset by peer
2020-12-15T08:59:05.614Z| mks| I125: SOCKET 4 (133) VNC Remote Disconnect.
vmkernel.log (we see the below vSCSI threads)
============
2020-12-15T09:58:12.261Z cpu13:55026677)VSCSI: 6726: handle 17032(vscsi0:0):Destroying Device for world 55026678 (pendCom 0)
2020-12-15T10:01:31.530Z cpu0:55115219)VSCSI: 6726: handle 17053(vscsi0:0):Destroying Device for world 55147988 (pendCom 0)
2020-12-15T10:01:32.482Z cpu14:55146528)VSCSI: 6726: handle 17047(vscsi0:0):Destroying Device for world 55146529 (pendCom 0)
2020-12-15T10:02:07.496Z cpu12:54963487)VSCSI: 6726: handle 17038(vscsi0:0):Destroying Device for world 55061792 (pendCom 0)
2020-12-15T10:03:01.379Z cpu21:55015068)VSCSI: 6726: handle 17043(vscsi0:0):Destroying Device for world 54982301 (pendCom 0)
2020-12-15T10:04:36.888Z cpu4:55148316)VSCSI: 6726: handle 17055(vscsi0:0):Destroying Device for world 55082781 (pendCom 0)
2020-12-15T10:04:54.174Z cpu7:54361224)VSCSI: 6726: handle 17052(vscsi0:0):Destroying Device for world 54426762 (pendCom 0)
2020-12-15T10:05:44.855Z cpu19:55146517)VSCSI: 6726: handle 17046(vscsi0:0):Destroying Device for world 55146518 (pendCom 0)
2020-12-15T10:09:57.690Z cpu15:55146820)VSCSI: 6726: handle 17049(vscsi0:0):Destroying Device for world 55146821 (pendCom 0
- Guests becomes slow and unresponsive.
- vmx dumps are created in the VM's directory:
vmx-zdump.001
vmx-zdump.002
vmx-zdump.003
vmx-zdump.002
vmx-zdump.003
- This primarily happens when the VM is on snapshots
vmware.log
=========
2020-12-15T07:45:08.263Z| vcpu-0| I125: Guest: toolbox-dnd: Version: 11.0.5.17716 (build-15389592)
2020-12-15T07:45:08.264Z| vcpu-1| W115: GuestRpc: application toolbox-dnd, changing channel 65535 -> 2
2020-12-15T07:45:08.264Z| vcpu-1| I125: GuestRpc: Channel 2, guest application toolbox-dnd.
2020-12-15T07:45:36.959Z| vmx| I125: GuestRpcSendTimedOut: message to toolbox-dnd timed out.
2020-12-15T07:45:36.959Z| vmx| I125: GuestRpcSendTimedOut: message to toolbox timed out.
2020-12-15T07:45:50.509Z| vcpu-0| I125: Tools: Tools heartbeat timeout.
2020-12-15T07:45:51.959Z| vmx| I125: GuestRpcSendTimedOut: message to toolbox-dnd timed out.
2020-12-15T07:45:51.959Z| vmx| I125: GuestRpc: app toolbox-dnd's second ping timeout; assuming app is down
2020-12-15T07:45:51.959Z| vmx| I125: GuestRpcSendTimedOut: message to toolbox timed out.
2020-12-15T07:45:51.959Z| vmx| I125: GuestRpc: app toolbox's second ping timeout; assuming app is down
2020-12-15T07:45:51.960Z| vmx| I125: GuestRpc: Reinitializing Channel 2(toolbox-dnd)
2020-12-15T07:45:51.960Z| vmx| I125: GuestMsg: Channel 2, Cannot unpost because the previous post is already completed
2020-12-15T07:45:51.960Z| vmx| I125: GuestRpc: Reinitializing Channel 0(toolbox)
2020-12-15T07:45:51.960Z| vmx| I125: GuestMsg: Channel 0, Cannot unpost because the previous post is already completed
2020-12-15T08:59:05.614Z| mks| I125: SSL: syscall error 104: Connection reset by peer
2020-12-15T08:59:05.614Z| mks| I125: SOCKET 4 (133) recv error 104: Connection reset by peer
2020-12-15T08:59:05.614Z| mks| I125: SOCKET 4 (133) VNC Remote Disconnect.
vmkernel.log (we see the below vSCSI threads)
============
2020-12-15T09:58:12.261Z cpu13:55026677)VSCSI: 6726: handle 17032(vscsi0:0):Destroying Device for world 55026678 (pendCom 0)
2020-12-15T10:01:31.530Z cpu0:55115219)VSCSI: 6726: handle 17053(vscsi0:0):Destroying Device for world 55147988 (pendCom 0)
2020-12-15T10:01:32.482Z cpu14:55146528)VSCSI: 6726: handle 17047(vscsi0:0):Destroying Device for world 55146529 (pendCom 0)
2020-12-15T10:02:07.496Z cpu12:54963487)VSCSI: 6726: handle 17038(vscsi0:0):Destroying Device for world 55061792 (pendCom 0)
2020-12-15T10:03:01.379Z cpu21:55015068)VSCSI: 6726: handle 17043(vscsi0:0):Destroying Device for world 54982301 (pendCom 0)
2020-12-15T10:04:36.888Z cpu4:55148316)VSCSI: 6726: handle 17055(vscsi0:0):Destroying Device for world 55082781 (pendCom 0)
2020-12-15T10:04:54.174Z cpu7:54361224)VSCSI: 6726: handle 17052(vscsi0:0):Destroying Device for world 54426762 (pendCom 0)
2020-12-15T10:05:44.855Z cpu19:55146517)VSCSI: 6726: handle 17046(vscsi0:0):Destroying Device for world 55146518 (pendCom 0)
2020-12-15T10:09:57.690Z cpu15:55146820)VSCSI: 6726: handle 17049(vscsi0:0):Destroying Device for world 55146821 (pendCom 0
Environment
VMware vSphere ESXi 6.7
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5
VMware vSphere ESXi 7.0.0
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5
Cause
Multiple VSCSI threads become stuck in the VSCSI layer which causes the VM to go into an unresponsive state.
Resolution
This issue is resolved in ESXi 6.7 P03 Build: 16713306 and 7.0 U1 Build: 16850804
Workaround:
No available workaround.
Workaround:
No available workaround.
Feedback
Yes
No
Powered by
