[VMC on AWS] VMDK Hot-extend operation and potential data loss

Products

• VMware Aria Suite
• VMware vCenter Server
• VMware Cloud on AWS

Issue/Introduction

To provide information on potential data loss after increasing the size of a VMDK using the Hot-Extend option.

Symptoms:
VMDK Hot-Extend operation was used to increase the size of a VMDK. 
The SDDC is version 1.24v1.
Storage vMotions or Virtual Machine restores using incremental CBT backups are being completed in the SDDC.

Environment

VMware vCenter Server 8.0.2

Cause

This is caused by a change in behavior in the VMDK Hot-extend operation that was introduced in SDDC versions 1.24v1. 
This can result in data loss when completing Storage vMotions and virtual machine restores with incremental CBT backup after the hot-extend.

Resolution

In order to resolve this issue, update to ESXi 8.0 Update 2b, build 23305546, or newer.

Workaround:
The recommendation is to only extend a VMDK when the virtual machine is in a powered-off state. 

If the Hot-Extend operation has already been completed:
1. Perform a virtual machine operation that will cause the VMDK to be internally re-opened. This includes power-on and power-off tasks, snapshot operations, suspending and resuming the virtual machine, or hot-removing and hot-adding the VMDK. 
2. Reset CBT and complete a full backup immediately. Note: This action does not fix the pre-existing backups. 

Additional Information

QueryChangedDiskAreas API returns incorrect sectors after extending virtual machine VMDK file with Changed Block Tracking (CBT) enabled in ESXi 8.0u2 (95940)

Migration of a powered-on VM after growing a disk can result in data corruption. (95958)

vSphere 8.0 U2 restored file corruption with resized VM disks

Resetting Changed Block Tracking for VMware vSphere virtual machines (2139574)

Impact/Risks:
Completing Storage vMotions or Virtual Machine restores with incremental CBT backups after the hot-extend operation will result in data loss.