In environments utilizing virtual machine backup applications such as Cohesity, incremental backups may fail to include data from one or more disks

Products

• VMware vCenter Server

Issue/Introduction

• Backup software such as Cohesity reports data has changed on affected disks, even when changes have occurred.

• Entire disks are skipped on the incremental backups for a VM.

• The issue affects multiple VMs across different ESXi hosts and clusters.

• Resetting CBT on the affected VMs temporarily resolves the issue, but it may reoccur.

Environment

• vSphere 8.x

Cause

The issue is often attributed to Changed Block Tracking (CBT) data becoming inconsistent or "corrupted," causing the `QueryAllocatedBlocks` API to return incorrect (empty) results to the backup application. 

However, investigation using VMware Virtual Disk Development Kit (VDDK) utilities (`vixDiskCheck` and `vixDiskLibSample`) confirm that the VMware API was correctly reporting allocated and changed blocks.  This suggests that the root cause may lie within environmental factors or third-party integrations, such as:

• Outdated firmware or software on the storage array (e.g., Pure Storage VVol).

• Outdated backup software versions.

Resolution

If you suspect CBT corruption is causing backup failures, follow these steps to troubleshoot and mitigate the issue:

• Reset CBT on affected VMs

  • Manually resetting CBT forces a recalculation of the block metadata.  Follow the steps in KB article 1020128 to perform a manual reset.

    • Changed Block Tracking (CBT) on virtual machines

• Verify CBT integrity with VDDK utilities

  • To determine if VMware is correctly tracking changes, use the Virtual Disk Development Kit (VDDK) utilities to sample the disk metadata before and after a backup failure:

    • **vixDiskCheck:** Use to check for data inconsistency in the CBT metadata.

      • Troubleshooting Changed Block Tracking (CBT)

    • **vixDiskLibSample:** Use with the `-getallocatedblocks` flag to verify if the API is returning 0 or valid allocated blocks.

• Update third-party infrastructure

  • Contact the backup application vendor to assist in troubleshooting (e.g. Cohesity, Commvault, Veeam, etc)

  • Ensure all components of the backup chain are running supported and updated versions, including the following: 

    • Backup application software

    • Storage arrays

Additional Information

• Virtual Disk Development Kit Release Notes

• VMware Virtual Disk Development Kit (VDDK)

• Troubleshooting Changed Block Tracking (CBT)