vSphere 8.0 U2 restored file corruption with resized VM disks

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

in vSphere 8.0U2 Changed Block Tracking (CBT) files may become inconsistent leading to backups not capturing the right data.

Symptoms may include:

• Inconsistencies or failures in backup restore processes.
• Discrepancies in data change tracking, especially post-disk resizing operations.
• Potential errors or warnings from backup solutions, particularly involving disk size changes.
• Data corruption of VM following powered-on vMotion (CPU or storage, per Migration of a powered-on VM after growing a disk can result in data corruption and [VMC on AWS] VMDK Hot-extend operation and potential data loss)

Environment

VMware vSphere ESXi 8.0.2

Cause

vSphere 8.0 Update 2 changed the way disks are extended. The changes were specifically intended to make certain disk hot-grow operations more efficient.

This has unintentionally resulted in incorrect tracking of changes, leading to backups not capturing the right data and, consequently, corrupt restores.

This issue will only be seen if backups are taken after the VM disk is hot extended. Simply resizing a VM with the VM powered off will not cause this.

This problem can happen with disks of all datastore types (VVOL, VMFS, NFS, vSAN).

Resolution

In order to resolve this issue, update to ESXi 8.0 Update 2b, build 23305546, or newer.

Workaround:

We strongly recommend to contact your backup software vendor to verify whether they have evolved steps to assist in recovering existing backups.

1. Grow disks only when the VM is powered off.
   • This is also called "cold-grow".
2. Perform an operation that will cause the disk to be internally re-opened, such as:
   • Power the affected VMs off, and then on again
   • Create a snapshot of the VM
   • Suspend, and then resume, the VM
   • hot-remove, and then hot-readd, the specific affected virtual disk.
3. Securing Future Backups:
   • While existing backups can't be retroactively fixed, future backups can be protected by resetting CBT on all vSphere VMs that had disk resizing post the upgrade to vSphere 8.0 U2.
   • Methods for CBT Reset:
     • a) Follow the VMware KB Article Resetting Changed Block Tracking for VMware vSphere virtual machines .
     • b) Perform an Active Full backup of the VMs with the backup software.

Additional Information

Migration of a powered-on VM after growing a disk can result in data corruption.

[VMC on AWS] VMDK Hot-extend operation and potential data loss

QueryChangedDiskAreas API returns incorrect sectors after extending virtual machine VMDK file with Changed Block Tracking (CBT) enabled in ESXi 8.0u2

Impact/Risks:

Powering VMs off and then back on again, hot-remove and hot-add of virtual disks, and suspending and resuming VMs, will cause the VM to be temporarily unavailable for production.

For some guest operating systems, creating a snapshot in virtual disks can fail, cause slowness or other problems. This particularly happens with very high input-output VMs such as database servers, and also with multi-writer servers, many of which are also often database servers.

For the CBT reset work-around, resetting CBT on VMs will cause the next backups to be full backups, which may require more resource bandwidth or time to process. These VMs may need their changes to be staggered to avoid oversaturation of the system, depending on your resources.