Multiple VMs experience latency issues.

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms : 

• Multiple virtual machines (VMs) experience latency issues across all hosts.

Validation Step: 

• In the var/run/log/vmkernel.log file, similar entries are seen:
  YYYY-MM-DDTHH:MM.SSSZ cpu1:2097886)WARNING: ScsiDeviceIO: 1513: Device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 396microseconds to 8037 microseconds.
YYYY-MM-DDTHH:MM.SSSZ cpu17:2097896)WARNING: ScsiDeviceIO: 1513: Device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 396 microseconds to 21192 microseconds.
YYYY-MM-DDTHH:MM.SSSZ cpu21:2097892)ScsiDeviceIO: 1513: Device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx performance has improved. I/O latency reduced from 21192 microseconds to 4179 microseconds.
YYYY-MM-DDTHH:MM.SSSZ cpu1:2097893)ScsiDeviceIO: 1513: Device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx performance has improved. I/O latency reduced from 4179 microseconds to 823 microseconds.
YYYY-MM-DDTHH:MM.SSSZ cpu6:2097895)WARNING: ScsiDeviceIO: 1513: naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 447 microseconds to 9825 microseconds.
YYYY-MM-DDTHH:MM.SSSZ cpu0:2097884)WARNING: ScsiDeviceIO: 1513: Device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 456 microseconds to 20588 microseconds.
  
  

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

Latency issues are caused by high storage utilization and increased DAVG (Device Average Response Time) values.

Cause validation: 

• Observe the DAVG values and the storage utilization using ESXTOP data.
  
  Please refer this KB : Using esxtop to identify storage performance issues for ESXi (multiple versions)
  
  
  
  
• From the above screenshot it is confirmed that,
  
  1) DAVG values (device average response time) are consistently high for the storage devices, which indicates that the storage devices are taking longer to respond to I/O requests.
  
  2) Also few storage devices are reaching 100% utilization, causing I/O operations to queue up and leading to these latency issues.

Resolution

• Engage storage vendor to investigate the high DAVG values reported in ESXTOP data.
  
  
• Engage the application team to identify any increased I/O generation and work on resolving the underlying cause.