High latency is observed on storage and virtual machines

Products

• VMware vCenter Server
• VMware vSphere ESXi 8.0
• VMware vSphere ESX 8.x
• VMware vSphere ESX 7.x
• VMware vSphere ESX 6.x

Issue/Introduction

• Multiple virtual machines (VMs) experience latency issues across all hosts.
• Under esxtop Storage Device Panel, high DAVG, KAVG and GAVG is observed
• Under esxtop Storage Device Panel (press 'u' to get storage devices), High number with QUED and 100% with %USD
• In the var/run/log/vmkernel.log file, LUN latency is observed similar to the entries listed below.
  YYYY-MM-DDTHH:MM.SSSZ cpu1:2097886)WARNING: ScsiDeviceIO: 1513: Device naa.#### performance has deteriorated. I/O latency increased from average value of 396microseconds to 8037 microseconds.
YYYY-MM-DDTHH:MM.SSSZ cpu17:2097896)WARNING: ScsiDeviceIO: 1513: Device naa.#### performance has deteriorated. I/O latency increased from average value of 396 microseconds to 21192 microseconds.
YYYY-MM-DDTHH:MM.SSSZ cpu21:2097892)ScsiDeviceIO: 1513: Device naa.#### performance has improved. I/O latency reduced from 21192 microseconds to 4179 microseconds.
YYYY-MM-DDTHH:MM.SSSZ cpu1:2097893)ScsiDeviceIO: 1513: Device naa.#### performance has improved. I/O latency reduced from 4179 microseconds to 823 microseconds.
YYYY-MM-DDTHH:MM.SSSZ cpu6:2097895)WARNING: ScsiDeviceIO: 1513: naa.#### performance has deteriorated. I/O latency increased from average value of 447 microseconds to 9825 microseconds.
YYYY-MM-DDTHH:MM.SSSZ cpu0:2097884)WARNING: ScsiDeviceIO: 1513: Device naa.#### performance has deteriorated. I/O latency increased from average value of 456 microseconds to 20588 microseconds.

Environment

VMware vSphere ESXi 8.x

Cause

Backend storage can not process the IO from ESX within the expected time.

Resolution

Engage storage vendor to investigate the high DAVG values reported in ESXTOP data.

Additional Information

• To understand storage utilization statistics under esxtop, please refer Storage Device Panel and refer Checking the queue depth of the storage adapter and the storage device to check storage utilization statistics.
• Please refer KB : Using esxtop to identify storage performance issues for ESXi for using esxtop to identify storage performance issues on ESXi