CBT reports larger area of changed blocks than expected if guest OS performed unmap on a disk

Products

• VMware vSphere ESXi
• VMware Cloud on AWS

Issue/Introduction

Symptoms:

• If a guest VM has disk unmap enabled, occasionally incremental backup takes longer time.
• If you check blocks that were backed up, you notice the total size of those blocks is close to the total disk size, regardless how many blocks were accurately modified.

Environment

VMware vSphere ESXi 6.7
VMware vSphere ESXi 6.5

Cause

When an unmap is triggered in the guest, the OS issues UNMAP requests to underlying storage. However, the requested blocks include not only unmapped blocks but also unallocated blocks. And all those blocks are captured by CBT and considered as changed blocks then returned to backup software upon calling the vSphere API queryChangedDiskAreas(changeId).

Resolution

To resolve this issue, upgrade to VMware VDDK 6.7 or later version.

Note: The backup software is able to filter out those unallocated blocks if it uses VDDK 6.7 or later. It can get all allocated area of a virtual disk by calling the VDDK API VixDiskLib_QueryAllocatedBlocks(), then taking the intersection with changed blocks reported by CBT. The result is the changes on the allocated blocks.
Storage system may allocate more space than needed by the guest OS. For example, VMFS file block size is 1MB and guest OS has a typical block size of 4KB. When 4KB data is requested to be written on the disk, 1 MB will be allocated and only 64KB will be recorded changed in CBT. VixDiskLib_QueryAllocatedBlocks()may report pre-allocated blocks on which nothing is yet written. It is expected that some allocated blocks may not be recorded in changed blocks.

Workaround:
To work around the issue, use one of these options:

• Disable unmap in guest VM. 
• If you have VDDK 6.7 or later libraries, take the intersection of VixDiskLib_QueryAllocatedBlocks() and queryChangedDiskAreas(changeId) to calculate the accurate changed blocks.