Database VMs report "Loss of Disk" or Guest OS I/O timeouts during backup window
search cancel

Database VMs report "Loss of Disk" or Guest OS I/O timeouts during backup window

book

Article ID: 444715

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • Database administrators (DBAs) report intermittent "loss of disk" errors or SCSI timeout events within high-transaction database VMs (e.g., SQL Server, Oracle).
  • The issue consistently occurs during scheduled backup windows or during snapshot consolidation tasks.
  • Virtual machines may become briefly unresponsive (pings dropped) for several seconds.
  • From the vmware.log from ssh session on the ESXi host the VM is currently running on, you observe "vm was stunned" entries as follows:

[from /vmfs/volumes/<datastore_name>/<vm_name>/vmware.log] 

####-##-##T##:##:##.###Z In(05) vcpu-0 - Checkpoint_Unstun: vm stopped for ######## us ####-##-##T##:##:##.###Z In(05) vcpu-0 - CPT: vm was stunned for ######## us

  • Performance returns to normal immediately after the snapshot consolidation process completes.

Environment

  • VMware vSphere ESXi (All versions)

Cause

This issue is caused by the VM Stun process. When a backup application triggers a snapshot consolidation, VMware briefly pauses (stuns) the VM to commit the delta data from the snapshot back to the base disk.

In high-I/O environments, the volume of data changed during the backup window (the "delta") can be large. Merging this large delta requires a longer stun period. If this pause exceeds the Guest OS or database application’s internal I/O timeout threshold, the system reports a loss of connectivity to the underlying disks.

Resolution

To mitigate extended stun times and prevent guest-level disk errors, implement the following best practices:

 

1. Increase Guest OS Disk Timeouts

Ensure the Guest OS is configured to tolerate brief I/O latencies.

    • For Windows: Verify the Disk/TimeOutValue registry setting is set to 60 seconds.
    • For Linux: Verify the udev timeout for SCSI disks is set to 60 or higher.

2. Consolidate Orphaned Snapshots

Leftover snapshots from failed backup attempts significantly increase the I/O overhead.

    • Right-click the VM in vSphere and select Snapshots > Consolidate.
    • Ensure the "Needs Consolidation" status is cleared in the VM summary tab.

3. Optimize the Backup Schedule

    • Avoid running backups during peak database maintenance windows (e.g., re-indexing, ETL jobs).
    • Distribute the backup start times for multiple database VMs to prevent simultaneous heavy I/O load on the same datastore/SAN.

4. Monitor Storage Latency
Check the ESXi host vmkernel.log from an ssh session on the host the VM is running on for storage performance warnings:

[From /var/log/vmkernel.log]

####-##-##T##:##:##.###Z cpu##:#######)WARNING: ScsiDeviceIO: ####: Device naa.################ performance has deteriorated. I/O latency increased from average value of #### microseconds to ###### microseconds. 


If these warnings are frequent, investigate the underlying SAN performance with your storage vendor.

5. Consider Non-Snapshot Backups for Ultra-High I/O VMs

If the database I/O is consistently too high for snapshots to consolidate within a safe window, consider using guest-based backup agents that do not rely on VMware-level snapshots.

Additional Information

Related Articles: 

Powered by Wolken Software