Datastore enters read-only state due to snapshot inflation during backups in vSphere
search cancel

Datastore enters read-only state due to snapshot inflation during backups in vSphere

book

Article ID: 449264

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Introduction A VMware vSphere datastore hosting virtual machines may reach 100% utilization and enter a read-only (RO) state during backup windows. This occurs when snapshot delta disks (SEsparse) grow significantly, consuming all available capacity before consolidation completes.

  • A datastore hosting a single Virtual Machine (VM) reaches 100% utilization and enters a read-only (RO) state.
  • Snapshot delta disks (SEsparse) grow significantly (e.g., several terabytes) during the backup window.
  • Snapshot consolidation takes an extended period (several hours) to complete.
  • vCenter tasks show successful snapshot creation and removal, but datastore space does not reclaim until consolidation finishes.

Environment

  • VMware vSphere ESXi 7.x / 8.x
  • Thin-provisioned virtual disks (VMDK)
  • Third-party snapshot-based backup software (e.g., Cohesity, Veeam, NetBackup)

Cause

The datastore exhaustion is caused by heavy write activity originating from within the Guest Operating System while a snapshot is active. VMware snapshots use a Copy-On-Write (COW) mechanism where any block modified by the Guest OS is written to the delta disk. High data churn (e.g., database maintenance or log rotations) during the backup window causes the delta disk to inflate.

Resolution

To resolve this issue, identify and mitigate the source of high data churn within the Guest OS:

  1. Review Guest OS Activity: Check Windows Event Logs, scheduled tasks, or database maintenance logs to identify processes writing large volumes of data during the backup window.
  2. Verify Hypervisor Logs: Examine the vmware.log for the affected VM to confirm the snapshot lifecycle.
  3. Rule out UNMAP Inflation: Run the following command to check for UNMAP/Retrim activity: grep -i "scsiX:Y" vmware*.log | grep -E -i "SEsparse|LBP|UNMAP" If no results return, the growth is confirmed to be active data writes.
  4. Transition to Agent-Based Backups: If high data churn is unavoidable, move the VM to an agent-based backup model to bypass hypervisor-level snapshots.
  5. Increase Capacity: Provision additional capacity to the datastore to provide a buffer for snapshot growth.
  6. Reschedule Workloads: Move heavy Guest OS tasks to occur outside of the backup window.
Powered by Wolken Software