VM's unresponsive due to underlying storage issue

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Multiple VM's might be unresponsive with underlying storage devices reporting high latency.
• Running esxtop on the ESXi through CLI reports High DAVG value indicating latency from the storage device (LUN):

 

• The var/run/log/vmkernel.log shows "performance has deteriorated" or "I/O latency increased"

[YYYY-MM-DDTHH:MM:SS] cpu51:2098041)WARNING: ScsiDeviceIO: 513: Device naa.60060060660########60606 performance has deteriorated. I/O latency increased from average value of 38762 microseconds to 776315 microseconds.
[YYYY-MM-DDTHH:MM:SS] cpu47:2098037)WARNING: ScsiDeviceIO: 1443: Device naa.60060060660########60606 performance has deteriorated. I/O latency increased from average value of 12017 microseconds to 254228 microseconds.
[YYYY-MM-DDTHH:MM:SS] cpu47:2098038)WARNING: ScsiDeviceIO: 1216: Device naa.60060060660########60606 performance has deteriorated. I/O latency increased from average value of 18057 microseconds to 534229 microseconds.

 

 

Environment

VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

This can be caused by insufficient available storage space in the storage array.

Resolution

Engage the storage vendor to investigate and remediate insufficient space issues on the storage array.

Additional Information

To get the LUN-level device performance statistics data, use the esxtop utility: Using esxtop to identify storage performance issues for ESXi (multiple versions)