vMotion of a virtual machine fails with the error: A general system error occurred: Failed to flush checkpoint data!
search cancel

vMotion of a virtual machine fails with the error: A general system error occurred: Failed to flush checkpoint data!

book

Article ID: 308477

calendar_today

Updated On: 11-04-2024

Products

VMware vSphere ESXi

Issue/Introduction

Symptoms:
  • Migrating virtual machines using vMotion, svMotion or xvMotion fails
  • You see the error:

    A general system error occurred: Failed to flush checkpoint data!
or

Failed waiting for data. Error 195887110. Limit exceeded.
  • The Error stack in the vCenter Server task details shows this message:
Migration [yyyyyyyyyyyy] failed. The checkpoint data length (xxxxx bytes) or the offset (xxxxxxxx bytes) exceeds the maximum checkpoint data length (xxxxxxxx byte).
  • This issue is more likely to occur if:
     
    • The resolution of the virtual machines is high or multiple screens are configured.
    • The virtual machine is using virtual machine hardware version below vmx-13 (ESXi 6.5 compatibility)
 
  • The virtual machine's vmware.log contains this error:

vmx| MigrateSetState: Transitioning from state 9 to 11.
vmx| Migrate_SetFailure: Failed waiting for data. Error bad0006. Limit exceeded.
vmx|
vmx| Migrate: cleaning up migration state.
vmx| MigrateSetState: Transitioning from state 11 to 0.
vmx| Msg_Post: Error
vmx| [vob.vmotion.chkpt.toobig] vMotion migration [XXXXXXXX:xxxxxxxxxxxxxxxx] failed. The checkpoint data length (xxxxx bytes) or the offset (xxxxxxxx bytes) exceeds the maximum checkpoint data length (xxxxxxxx byte).
vmx| [msg.moduletable.powerOnFailed] Module Migrate power on failed.


or

vcpu-0| I125: Msg_Post: Error
vcpu-0| I125: [msg.checkpoint.migration.writefail] Failed to write checkpoint data (offset 25151361, size 16384): Limit exceeded.
vcpu-0| I125: ----------------------------------------
Worker#0| I125: SVMotionCleanupThread: Waiting for SVMotion Bitmap thread to complete.
Worker#1| W115: SVMotionThreadCompleteMigration: Failed while informing vmkernel of disk close: Migration determined a failure by the VMX
vcpu-0| W115: MigrateWrite: failed: Limit exceeded
vcpu-0| I125: MigrateDumperWrite: MigrateWrite failed
(…)
vcpu-0| I125: Msg_Reset:
vcpu-0| I125: [msg.checkpoint.save.fail.ioError] An error occurred


Cause

The default pre-allocated buffer may be too small for certain virtual machines with higher resolutions.

Resolution

You can try one of the below options to address this issue.
  • Decrease the VM’s resolution before using vMotion for migration.
  • Make sure virtual machine is running with Virtual Hardware Version 13 or above.
  • Increase the base checkpoint cache size. Doubling this value from its default of 8 MB to 16 MB (16777216 bytes) should be enough for most common display resolutions. Note that this might not be sufficient for multiple displays / very large resolution.
    • To increase the base checkpoint cache size:
      1. Power off the virtual machine.
      2. Click the virtual machine in the Inventory.
      3. On the Summary tab for that virtual machine, click Edit Settings.
      4. In the virtual machine Properties dialog box, click the Options tab.
      5. Under Advanced, select General and click Configuration Parameters.
      6. Click Add Row.
      7. In the new row, add migrate.baseCptCacheSize to the name column and add 16777216 to the value column.
      8. Click OK to save the change.

Alternatively, migrate.baseCptCacheSize can be set for powered on virtual machines using PowerCLI. The following commands would set the VMX option without downtime. When migrating the virtual machine, the new VMX process is created on the target host and the buffer is set to the size defined with the option, so no vMotion failure occurs:

$vms = Get-VM (define your list of virtual machines here) | Get-View
$vmx = New-Object VMware.Vim.VirtualMachineConfigSpec
$vmx.extraConfig += New-Object VMware.Vim.OptionValue
$vmx.extraConfig[0].key = "migrate.baseCptCacheSize"
$vmx.extraConfig[0].value = "16777216"
($vms).ReconfigVM_Task($vmx)


Note: You might need to increase the value for VMs with larger resolutions, do so at least in 2 MB increments (MB x 1024 x 1024). The overhead of this increase will be limited to the duration of the vMotion.