Snapshot consolidation takes a long time - times out after a few hours.

Products

• VMware vSphere ESXi

Issue/Introduction

• Regular VM snapshot is taking long time to consolidate.
• Snapshot consolidation that should take minutes times out after 2 hours. 
• Production VM performance significantly degraded even before snapshot consolidation initiated. 

Environment

VMware vSpehre ESXi 8.x

Cause

• There is extreme latency on the LUN on which the affected VM resides

 

• /var/log/vmkernel.log   or /var/log/vmkwarning.log may report logging similar to:

 

[YYYY-MM-DDTHH:MM:SS] cpu51:2098041)WARNING: ScsiDeviceIO: 513: Device naa.########## performance has deteriorated. I/O latency increased from average value of 38762 microseconds to 776315 microseconds.
[YYYY-MM-DDTHH:MM:SS] cpu47:2098037)WARNING: ScsiDeviceIO: 1443: Device naa.######### performance has deteriorated. I/O latency increased from average value of 12017 microseconds to 254228 microseconds.
[YYYY-MM-DDTHH:MM:SS] cpu47:2098038)WARNING: ScsiDeviceIO: 1216: Device naa.######### performance has deteriorated. I/O latency increased from average value of 18057 microseconds to 534229 microseconds.

 

• esxtop reports high DAVG/cmd values on the LUN. DAVG/cmd is the average response time in milliseconds per command being sent to the device. For more information see Using esxtop to identify storage performance issues for ESXi
  
  

Resolution

Engage your storage vendor and fabric/network team to investigate the cause of the latency.