QueryChangedDiskAreas API returns incorrect sectors after extending virtual machine VMDK file with Changed Block Tracking (CBT) enabled
search cancel

QueryChangedDiskAreas API returns incorrect sectors after extending virtual machine VMDK file with Changed Block Tracking (CBT) enabled

book

Article ID: 310751

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
When using backup software that uses the Virtual Disk Development Kit (VDDK) API call QueryChangedDiskAreas() to return a list of allocated disk sectors, you experience these symptoms:
  • The list of allocated virtual machine disk sectors returned is incorrect.
  • Backups appear to be corrupt or missing data.


Environment

VMware vSphere ESXi 5.0
VMware vSphere ESXi 5.5
VMware vSphere ESXi 5.1
VMware ESXi 4.0.x Installable
VMware ESXi 4.1.x Installable
VMware ESXi 4.1.x Embedded

Cause

The Virtual Disk Development Kit (VDDK) method QueryChangedDiskAreas() is used by some backup applications to determine which areas of a virtual disk have changed since the last backup, this information is used to decide which blocks to include in a backup.

This issue occurs when you expand a virtual disk .vmdk file which has Change Block Tracking (CBT) enabled past any 128 GB boundary. In ESXi 4.x and earlier versions of ESXi 5.x when the disk is extended, the change tracking data becomes unreliable. Due to the faulty changed block information, some changed blocks might not be captured by a backup. Consequently, a restore operation that uses the incomplete backup could cause a data loss.

Resolution

This is a known issue affecting VMware ESXi 4.x and ESXi 5.x.

This issue is resolved in these releases:

Currently, there is no resolution for ESXi 4.x.

To work around this issue, you must disable CBT and then enable it for disks extended past a 128 GB boundary. You can do this through a vCenter Client only on a powered off virtual machine.
To disable CBT and then enable it for disks extended past a 128 GB boundary using the vCenter Client, perform these steps:
  1. Power off the virtual machine.
  2. Use a vCenter client to disable CBT on each disk extended past a 128 GB boundary.
  3. Power on the virtual machine.
  4. Power off the virtual machine.
  5. Use a vCenter client to enable CBT on each disk where you disabled it in step 2.
  6. Power on the virtual machine.

To work around this issue where the virtual machine must remain powered on, perform these steps:

  1. Disable CBT using either PowerCLI or the VDDK API.
  2. Take a snapshot.
  3. Delete the snapshot (To recover space and performance).
  4. Enable CBT using either PowerCLI or the VDDK API.
  5. Take a snapshot.
  6. Delete the snapshot.
The next backup after toggling CBT is the full backup of the virtual machine.

Note: Discard any backups that were captured after growing the disk, as they can be incomplete.



Additional Information

For more information about the VDDK API call QueryChangedDiskAreas(), see the vSphere 5.0 SDK documentation and vSphere 5.1 SDK documentation.

Frequently Asked Questions:

Is the fix retroactive on corrupted .ctk files that already exist?
Basically, the answer is no. If the .ctk files are corrupted the way forward is to trigger a full backup using the methods in the KB. Since making the fix retroactive would require repairing .ctk files and since the data needed to do so is unavailable, repairing suspect backups (or the .ctk files they are based on) is not an available option.

Is there a way to determine if a virtual disk has been expanded?
You should rely on your own change control records to determine whether a virtual disk has been expanded. This information is not tracked in the virtual machine.

Is virtual machine growth in smaller increments affected?
The amount of space the virtual disk is extended is not relevant, the increment of space by which a virtual disk is extended is not relevant. The virtual machine has this issue when its disk is grown past any 128 GB boundary in absolute size. The issue is triggered at other sizes which are a power of 2 from 128 GB up. For example: 256 GB, 512 GB, and 1024 GB.

Check whether CBT is enabled
  • In the home directory of the virtual machine, verify whether a vmname-ctk.vmdk file is saved for one or more virtual hard disks.
  • To Query the advanced configuration parameters for the virtual machine:
    1. Right-click the virtual machine and click Edit Settings.
    2. Click the Options tab.
    3. Click General in the Advanced section and then click Configuration Parameters. The Configuration Parameters dialog opens.
    4. Search for the ctkEnabled parameter entry for each disk and note whether it is enabled.
Note: The VDDK API call configSpec.changeTrackingEnabled = new Boolean(true); can enable and disable CBT dynamically. In this case, you need not power off the virtual machine to change the CBT configuration as taking and deleting a snapshot can apply these settings.

Important: The use of both the VDDK API and scripts is only supported with VMware SDK Developer Support. For more information, see the API 5.5 Documentation and VMware SDK Developer Support Program (1029840).
Applying the fix
Using the ExtendVirtualDisk function to grow virtual disks is not an ESX host or a virtual machine specific operation. The vCenter Server can use any host that has access to the datastore for growing a virtual disk. Therefore, you must apply the above resolution to all ESX hosts with access to the datastore managed by vCenter Server.

Changed Block Tracking (CBT) on virtual machines
VMware SDK Developer Support Program
Enabling or disabling Changed Block Tracking (CBT) on virtual machines
VMware ESXi 5.0, Patch Release ESXi500-201412001
変更ブロックのトラッキング (CBT) を有効にして仮想マシン VMDK ファイルを拡張した後に QueryChangedDiskAreas API が正しくないセクタを返す
VMware ESXi 5.5, Patch Release ESXi550-201501001
QueryChangedDiskAreas API 扩展已启用更改块跟踪 (CBT) 的虚拟机 VMDK 文件后返回不正确的扇区

Powered by Wolken Software