Changed Block Tracking (CBT) is inconsistent after resizing VM disk in vSphere 8.0 U2

Products

• VMware vSphere ESXi

Issue/Introduction

In vSphere 8.0U2 Changed Block Tracking (CBT) files may become inconsistent leading to backups not capturing the right data which could result in:

• Inconsistencies or failures in backup restore processes.
• Discrepancies in data change tracking, especially post-disk resizing operations.
• Potential errors or warnings from backup solutions, particularly involving disk size changes.
• Data corruption of VM following powered-on vMotion (CPU or storage, per Migration of a powered-on VM after growing a disk can result in data corruption. and [VMC on AWS] VMDK Hot-extend operation and potential data loss

Environment

VMware vSphere ESXi 8.0U2

Cause

vSphere 8.0 Update 2 changed the way disks are extended. The changes were specifically intended to make certain disk hot-grow operations more efficient.

This has unintentionally resulted in incorrect tracking of changes, leading to backups not capturing the right data and, consequently, corrupt restores.

This issue will only be seen if backups are taken after the VM disk is hot extended. Simply resizing a VM with the VM powered off will not cause this.

This problem can happen with disks of all datastore types (VVOL, VMFS, NFS, vSAN).

Resolution

This issue is fixed in ESXi 8.0U2b build 23305546 or later.

Workaround:

We strongly recommend contacting the backup software vendor to verify whether they have evolved steps to assist in recovering existing backups.

1. Grow disks only when the VM is powered off.
   • This is also called "cold grow".
2. Perform an operation that will cause the disk to be internally re-opened, such as:
   • Power Cycle the affected VMs.
   • Create a snapshot of the affected VM.
   • Suspend and resume the affected VM.
   • Remove and hot-add the specific affected virtual disk.
3. Securing Future Backups:
   • While existing backups can't be retroactively fixed, future backups can be protected by resetting CBT on all vSphere VMs that had disk resizing post the upgrade to vSphere 8.0 U2.
   • Methods for CBT Reset:
     • Resetting Changed Block Tracking for VMware vSphere virtual machines.
     • Perform an Active full backup of the VMs with the backup software.

Additional Information

Impact/Risks:

Powering off the VM, removing and re-adding virtual disks, suspending and resuming VM will cause the VM to be temporarily unavailable for production.

For some guest operating systems, creating a snapshot in virtual disks can fail, cause slowness or other problems. This happens particularly with very high input-output VMs such as database servers, multi-writer servers many of which are often database servers.

Resetting CBT on VMs will cause the next backups to be full backups, which requires more resource, bandwidth and time to complete.