Remove snapshot very slow and timeout finally.

Products

• VMware vSphere ESXi 8.0

Issue/Introduction

• The snapshot removal task is very slow.
• In the var/run/log/vmkernel.log file, similar entries are seen:
  YYYY-MM-DDTHH:MM.SSSZ cpux:2097xxx)WARNING: ScsiDeviceIO: 1780: Device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 896 microseconds to 127019 microseconds.
  YYYY-MM-DDTHH:MM.SSSZ cpux:2097xxx)WARNING: ScsiDeviceIO: 1780: Device naa.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx performance has deteriorated. I/O latency increased from average value of 687 microseconds to 107247 microseconds.
• Output of esxtop command shows device with high DAVG and KAVG, and 100 %USD.

Environment

VMware vSphere ESXi 8.x

Cause

Performance issue is caused by high storage utilization and increased DAVG (Device Average Response Time) values.

Storage device is reaching 100% utilization, causing I/O operations to queue up and leading to these latency issues.

Resolution

• Engage storage vendor to investigate the high DAVG values reported in ESXTOP data.
• Engage the application team to identify any increased I/O generation and work on resolving the underlying cause.

Additional Information

Using esxtop to identify storage performance issues for ESXi (multiple versions)