Backing up the virtual machine fails when CBT is enabled

Products

• VMware vSphere ESXi

Issue/Introduction

To resolve the issue when backing up the virtual machine fails if CBT is enabled, upgrade to ESXi 6.0 Build 2715440.

Symptoms:

• Backing up a virtual machine with Changed Block Tracking (CBT) enabled fails after upgrading to VMware ESXi 6.0.x on host
• Backing up a virtual machine with Changed Block Tracking (CBT) enabled fails after installing VMware ESXi 6.0.x on host
• Powering on the virtual machines fails.
• Expanding the size of a virtual disk fails.
• Taking virtual machine quiesced snapshots fails.
• The vSphere Client displays the error similar to:
  
  An error occurred while taking a snapshot: msg.snapshot.error-QUIESCINGERROR
  
  Note: This error may or may not be present

• In the /var/log/vmkernel.log file on the ESXi host where the affected virtual machine is running, you see the error similar to:

  <YYYY-MM-DD>T<time>.623Z cpu5:809536)WARNING: CBT: 191: No memory available! Called from 0x4180219af50e
  <YYYY-MM-DD>T<time>.637Z cpu5:809536)WARNING: CBT: 191: No memory available! Called from 0x4180219af50e
  <YYYY-MM-DD>T<time>.648Z cpu5:809536)WARNING: CBT: 191: No memory available! Called from 0x4180219af50e</time></time></time>
  
  Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

• In the vmware.log file of the affected virtual machine, you see an entry similar to:
  
  vcpu-0| I120: DISKLIB-CBT : Creating cbt node 92b78c-cbt failed with error Cannot allocate memory (0xbad0014, Out of memory)

Environment

VMware vSphere ESXi 6.0

Cause

This issue occurs due to heap exhaustion. It can occur when attempting to enable Changed Block Tracking (CBT). If a virtual machine with a large number of virtual disks reached an upper threshold, enabling CBT fails because of heap exhaustion. This issue also occurs with multiple virtual machines with CBT enabled. In case of Windows virtual machines with VSS enabled, taking a quiesced snapshot creates double the amount of memory overhead. Finally, if the heap is close to exhaustion, performing a vMotion can also cause this issue as that process involves taking snapshots as well.

Note: The virtual disks can be spread across virtual machines or can be in a single virtual machine.

Resolution

This issue is resolved in VMware ESXi 6.0 Build 2715440, available at https://support.broadcom.com

Additional Information

For more information on enabling and disabling CBT, see:

Enabling Changed Block Tracking (CBT) on virtual machines
Changed Block Tracking (CBT) on virtual machines