VM become unresponsive and unable to launch Web Console

Products

• VMware vSphere ESXi

Issue/Introduction

Several VMs frequently become unresponsive and the Web Console cannot be launched.

In vmware.log, the GuestRpcSendTimedOut message to toolbox is continuously being recorded.

YYYY-MM-DDTXX:XX:XX.XXX In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
YYYY-MM-DDTXX:XX:XX.XXX In(05) vmx - Tools: [AppStatus] Last heartbeat value 326955 (last received 9s ago)
YYYY-MM-DDTXX:XX:XX.XXX In(05) vmx - TOOLS: appName=toolbox, oldStatus=1, status=2, guestInitiated=0.
YYYY-MM-DDTXX:XX:XX.XXX In(05) vmx - GuestRpcSendTimedOut: message to toolbox timed out.
YYYY-MM-DDTXX:XX:XX.XXX In(05) vmx - GuestRpc: app toolbox's second ping timeout; assuming app is down
YYYY-MM-DDTXX:XX:XX.XXX In(05) vmx - Tools: [AppStatus] Last heartbeat value 326955 (last received 29s ago)
YYYY-MM-DDTXX:XX:XX.XXX In(05) vmx - TOOLS: appName=toolbox, oldStatus=2, status=0, guestInitiated=0.

 

At the same time, there are instances of storage errors and performance degradation in vmkernel.log. This datastore is where the VM located.

YYYY-MM-DDTXX:XX:XX.XXX In(182) vmkernel: cpu97:2098613)ScsiDeviceIO: 4605: Cmd(0x45da2de3c100) 0x8a, cmdId.initiator=0x430f2f017240 CmdSN 0x3ff from world 25436963 to dev "naa.xxxxxxxxxxxxxxxxxx" failed H:0x8 D:0x0 P:0x0 Cancelled from driver layer 
YYYY-MM-DDTXX:XX:XX.XXX In(182) vmkernel: cpu52:2097363)NMP: nmp_ThrottleLogForDevice:3842: last error status from device naa.xxxxxxxxxxxxxxxxxx repeated 20 times
YYYY-MM-DDTXX:XX:XX.XXX In(182) vmkernel: cpu52:2097363)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0xa3 (0x45da7ed50200, 0) to dev "naa.xxxxxxxxxxxxxxxxxx" on path "vmhba4:C0:T6:L0" Failed: 
YYYY-MM-DDTXX:XX:XX.XXX In(182) vmkernel: cpu2:2098591)NMP: nmp_ThrottleLogForDevice:3893: Cmd 0xf1 (0x45babab4d7c0, 25980411) to dev "naa.xxxxxxxxxxxxxxxxxx" on path "vmhba3:C0:T4:L0" Failed: 
YYYY-MM-DDTXX:XX:XX.XXX In(182) vmkernel: cpu8:2098585)ScsiDeviceIO: 4633: Cmd(0x45ba40f77080) 0xfe, CmdSN 0x663d34c2 from world 25980415 to dev "naa.xxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x5  
YYYY-MM-DDTXX:XX:XX.XXX In(182) vmkernel: cpu102:2098582)ScsiDeviceIO: 4633: Cmd(0x45da73dd2900) 0xfe, CmdSN 0x663d34c6 from world 25980408 to dev "naa.xxxxxxxxxxxxxxxxxx" failed H:0x0 D:0x2 P:0x5

……

YYYY-MM-DDTXX:XX:XX.XXX Wa(180) vmkwarning: cpu46:2098568)WARNING: ScsiDeviceIO: 1780: Device naa.xxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 2117 microseconds to 62825 microseconds.

YYYY-MM-DDTXX:XX:XX.XXX Wa(180) vmkwarning: cpu46:2098568)WARNING: ScsiDeviceIO: 1780: Device naa.xxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 2117 microseconds to 62825 microseconds.
YYYY-MM-DDTXX:XX:XX.XXX Wa(180) vmkwarning: cpu46:2098574)WARNING: ScsiDeviceIO: 1780: Device naa.xxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 2118 microseconds to 60798 microseconds.

Environment

vSphere vCenter Server 8.x

Cause

Devices are reporting high latency, causing virtual machines to become unresponsive during I/O operations.

Resolution

Contact the storage vendor to fix the storage performance degradation issue.

For example, the performance of some EMC storage systems can significantly decrease when there is insufficient available space.

Additional Information

ESXi device latency with "performance has deteriorated" messages in ESXi host logs