Understanding Multi-Writer Disks and Their Relation to Snapshots and Backups

Products

VMware vCenter Server VMware vCenter Server 6.0 VMware vCenter Server 7.0 VMware vCenter Server 8.0 VMware vSphere ESXi 6.0 VMware vSphere ESXi 7.0 VMware vSphere ESXi 8.0

Issue/Introduction

This article provides an insight on :

Snapshot Behavior for Multi-Writer Disks
Why Snapshots Are Not Recommended for Multi-Writer Disks?
Impact of Disk Locks and I/O Operations During Snapshots on Multi-Writer Disks
Snapshots and VMDKs Larger Than 2TB in Multi-Writer Configurations

Symptoms:
When backing up a virtual machine, the following symptoms are observed.

The backup of the virtual machine fails.
- Snapshot backup failing for the VMs with multi-writer VMDKs with CBT error.

- ERROR: CBT-enabled VM having multi-writer disk(s) is not supported.

Environment

VMware vSphere ESXi 6.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 8.x

Cause

Multi-writer snapshot supported and unsupported actions or features:

Actions or Features	Supported	Unsupported	Notes
Snapshots of VMs with dependent disks		×	Virtual backup solutions leverage snapshots through the vStorage APIs; for example, VMware Data Recovery, vSphere Data Protection. These are also not supported.
Snapshots of VMs with independent-persistent disks	√		Supported in vSphere 5.1 update2 and later versions. The snapshot task will be completed successfully but will skip the independent disks from the snapshot operation. Powered on virtual machines with independent disks cannot support memory snapshots.
Changed Block Tracking (CBT)		×

Resolution

What Are Multi-Writer Disks?

Multi-writer disks are virtual disks configured to allow simultaneous read/write access by multiple virtual machines (VMs). This configuration is commonly used in scenarios like clustering, distributed databases, and applications requiring shared storage. Multi-writer disks are ideal for environments where data consistency and performance are critical across multiple VMs.

VMFS is a clustered file system that disables (by default) multiple virtual machines from opening and writing to the same virtual disk (.vmdk file). This prevents more than one virtual machine from inadvertently accessing the same .vmdk file.

The multi-writer option allows VMFS-backed disks to be shared by multiple virtual machines. This option is used to support VMware fault tolerance, which allows a primary virtual machine and a standby virtual machine to simultaneously access a .vmdk file.

Snapshot Behavior for Multi-Writer Disks

Dependent Mode Multi-Writer Disks:
Snapshots are not supported for dependent mode disks with the multi-writer flag. Dependent disks are typically included in snapshots, but multi-writer configurations disable this functionality due to the inability to handle redo logs (delta files) in shared-disk environments. This restriction prevents data corruption caused by conflicting writes.
Independent Mode Multi-Writer Disks:
- Independent - Persistent Mode: Snapshots do not affect disks in this mode. Changes are written directly to the base disk and cannot be reverted.
- Independent - Nonpersistent Mode: Snapshots do not include disks in this mode. Changes are temporary and discarded when the VM is powered off or reset. Backup operations relying on snapshots exclude these disks entirely.

Impact of Disk Locks and I/O Operations During Snapshots

When a snapshot is created, VMware temporarily suspends I/O operations to the disk to establish a consistent state. A redo log (delta file) is created to track changes, while the base disk becomes read-only. For multi-writer disks, this mechanism introduces the following challenges:

Disk Locks: Shared-disk environments rely on simultaneous read/write access by multiple VMs. Snapshot operations impose exclusive locks on disks to manage delta files, which disrupt shared access and cause failures or data inconsistencies.
I/O Redirection: During a snapshot, all I/O operations are redirected to the delta file. Multi-writer applications, such as databases or clustered file systems, cannot efficiently handle this redirection, leading to performance degradation and potential corruption.

These inherent limitations of the snapshot mechanism make it unsuitable for multi-writer disks.

Snapshots and VMDKs Larger Than 2TB in Multi-Writer Configurations

Multi-writer disks larger than 2TB do not support the VMFSSPARSE snapshot format. This is because VMFSSPARSE has a size limitation that does not extend to disks larger than 2TB. For disks larger than 2TB, snapshots automatically use the SESPARSE format instead of VMFSSPARSE. However, enabling multi-writer mode on a disk disables snapshot functionality entirely, regardless of whether SESPARSE could theoretically be used.

Here’s a detailed breakdown:

1. VMFSSPARSE Format Limitations:

VMFSSPARSE is the older snapshot format.
It cannot handle VMDKs larger than 2TB due to architectural constraints.
As of vSphere 5.5 and later, SESPARSE was introduced specifically to address this limitation, allowing snapshots of VMDKs larger than 2TB.

2. SESPARSE Format Enhancements:

SESPARSE supports VMDKs larger than 2TB.
Automatically replaces VMFSSPARSE for snapshot redo logs when the base disk is larger than 2TB.
However, SESPARSE does not override the restriction on snapshots for multi-writer disks.

3. Multi-Writer Disk Restriction:

Multi-writer mode disables snapshots to avoid conflicts and inconsistencies during simultaneous writes by multiple VMs.
This applies regardless of the disk size or snapshot format.

Changed Block Tracking (CBT)

Normal Disks: CBT is supported on non-multi-writer disks and allows incremental backups by tracking changes in the virtual disk.
Multi-Writer Disks: CBT does not work for multi-writer disks because of the complexity introduced by simultaneous writes from multiple VMs

Backup Types and Multi-Writer Disks

Snapshot-Based Backups
- Relies on VMware snapshots to capture VM states.
- Incompatibility: As snapshots are unsupported for multi-writer disks, this method is not viable.
Agent-Based Backups
- Uses backup agents installed on the guest OS to back up files or application data directly.
- Advantages: Works independently of VMware snapshots, making it compatible with multi-writer disks.

Why Snapshot-Based Backup Is Not Supported for Multi-Writer Disks

Redo Log Limitations: Redo logs cannot manage concurrent write operations required by multi-writer disks.
Disk Locks: Locks conflict with the shared access needed for multi-writer functionality.
Data Integrity Risks: Snapshot mechanisms may introduce inconsistencies or corruption in multi-writer environments.
Performance Impact: Snapshots redirect I/O operations, which significantly degrades performance in shared-disk configurations