Backing up a Changed Block Tracking enabled virtual machine in ESXi 6.0.x returns incorrect changed sectors

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

When running virtual machine backups which utilize Changed Block Tracking (CBT) in ESXi 6.0, you experience these symptoms:

• The CBT API call QueryDiskChangedAreas()can sometimes return incorrect changed sectors, which results in inconsistent incremental virtual machine backups.
• Inconsistent virtual machine backups.

Environment

VMware vSphere ESXi 6.0

Cause

When running incremental virtual machine backups, backup applications typically rely on the vSphere API call QueryDiskChangedAreas() to determine the changed sectors.

 

This issue occurs due to a problem with CBT in the disklib area, which results in the change tracking information of I/Os that occur during snapshot consolidation to be lost. The main backup payload data is never lost and it is always written to the backend device. However, the corresponding change tracking information entries which occur during the consolidation task are missed. Subsequent QueryDiskChangedAreas() calls do not include these missed blocks and, therefore a backup based on this CBT data is inconsistent.

 

All incremental backups which utilize CBT are potentially affected.

Resolution

This issue is resolved in VMware ESXi 6.0 patch ESXi600-201511001. For more information, see VMware ESXi 6.0, Patch Release ESXi600-201511001 (2137545).

 

Required steps to ensure incremental virtual machine backup integrity

 

After the host upgrade is complete, you must create a full virtual machine backup to ensure incremental virtual machine backup integrity. All subsequent incremental backups must reference only the change IDs generated since the full backup (post patch install). You must ensure that the change IDs generated before the host upgrade are not referenced, as this can lead to incorrect sectors being returned by CBT.

Important considerations:

• If you are using Eager Zeroed thick virtual disks, just a full virtual machine backup is sufficient to resume incremental virtual machine backups.
• If you are using other types of virtual disks, a full reset of CBT data is required, see Resetting Changed Block Tracking for VMware vSphere virtual machines (2139574).

For more information about supported virtual disk types, see Types of supported Virtual Disks on ESXi/ESX hosts (1022242).

 

If you are unable to upgrade, use one</u> of these options to work around this issue:

• Downgrade the affected ESXi hosts to version 5.5, and downgrade the virtual Hardware Version from 11 to 10, if necessary. For more information, see:
  • Reverting to a previous version of ESXi (1033604)
  • Downgrading the virtual machine hardware version in ESX/ESXi (1028019)
     
• Shutdown the virtual machine before doing an incremental backup.
   
• Do a full virtual machine backup rather than an incremental backup.

Note: Your virtual machine backup solution vendor may have additional information about the impact of this issue on your backup solution, and specific steps that can be applied to mitigate the issue. For more information, contact your backup vendor's support.

 

Additional Information

 

Types of supported Virtual Disks on ESXi/ESX hosts
Downgrading the virtual machine hardware version in ESX/ESXi
Reverting to a previous version of ESXi
VMware ESXi 6.0, Patch Release ESXi600-201511001
在 ESXi 6.0.x 中备份启用了更改块跟踪的虚拟机时返回错误的更改扇区
ESXi 6.0.x で Changed Block Tracking 機能を有効にした仮想マシンをバックアップすると、不正な変更済みセクタが返される
Resetting Changed Block Tracking for VMware vSphere virtual machines
La copia de seguridad de una máquina virtual habilitada para Seguimiento de bloqueo cambiado en ESXi 6.0.x devuelve sectores cambiados incorrectos
O backup de uma máquina virtual habilitada com Rastreamento de Bloco Alterado no ESXi 6.0.x retorna setores alterados incorretos

Impact/Risks:
Data loss could occur on restore from an inconsistent virtual machine backup.