vMotion of a virtual machine fails with the error: A general system error occurred: Failed to flush checkpoint data!
Article ID: 308477
Updated On:
Products
Issue/Introduction
- Migrating virtual machines using vMotion, svMotion or xvMotion fails
- You see the error:
A general system error occurred: Failed to flush checkpoint data!
Failed waiting for data. Error 195887110. Limit exceeded.- The Error stack in the vCenter Server task details shows this message:
Migration [yyyyyyyyyyyy] failed. The checkpoint data length (xxxxx bytes) or the offset (xxxxxxxx bytes) exceeds the maximum checkpoint data length (xxxxxxxx byte).- This issue is more likely to occur if:
- The resolution of the virtual machines is high or multiple screens are configured.
- The virtual machine is using virtual machine hardware version below vmx-13 (ESXi 6.5 compatibility)
- The virtual machine's
vmware.logcontains this error:
vmx| MigrateSetState: Transitioning from state 9 to 11.
vmx| Migrate_SetFailure: Failed waiting for data. Error bad0006. Limit exceeded.
vmx|
vmx| Migrate: cleaning up migration state.
vmx| MigrateSetState: Transitioning from state 11 to 0.
vmx| Msg_Post: Error
vmx| [vob.vmotion.chkpt.toobig] vMotion migration [XXXXXXXX:xxxxxxxxxxxxxxxx] failed. The checkpoint data length (xxxxx bytes) or the offset (xxxxxxxx bytes) exceeds the maximum checkpoint data length (xxxxxxxx byte).
vmx| [msg.moduletable.powerOnFailed] Module Migrate power on failed.
or
vcpu-0| I125: Msg_Post: Error
vcpu-0| I125: [msg.checkpoint.migration.writefail] Failed to write checkpoint data (offset 25151361, size 16384): Limit exceeded.
vcpu-0| I125: ----------------------------------------
Worker#0| I125: SVMotionCleanupThread: Waiting for SVMotion Bitmap thread to complete.
Worker#1| W115: SVMotionThreadCompleteMigration: Failed while informing vmkernel of disk close: Migration determined a failure by the VMX
vcpu-0| W115: MigrateWrite: failed: Limit exceeded
vcpu-0| I125: MigrateDumperWrite: MigrateWrite failed
(…)
vcpu-0| I125: Msg_Reset:
vcpu-0| I125: [msg.checkpoint.save.fail.ioError] An error occurred
Environment
VMware vSphere ESXi 6.5
VMware vSphere ESXi 6.7
Cause
Resolution
Workaround:
• Decrease the VM’s resolution before using vMotion for migration.
• Upgrade to Virtual Machine Hardware version 13 if below.
• Increase the base checkpoint cache size. Doubling this value from its default of 8 MB to 16 MB (16777216 bytes) should be enough for most common display resolutions. Note that this might not be sufficient for multiple displays / very large resolutions.
To increase the base checkpoint cache size:
- Power off the virtual machine.
- Click the virtual machine in the Inventory.
- On the Summary tab for that virtual machine, click Edit Settings.
- In the virtual machine Properties dialog box, click the Options tab.
- Under Advanced, select General and click Configuration Parameters.
- Click Add Row.
- In the new row, add
migrate.baseCptCacheSizeto the name column and add16777216to the value column. - Click OK to save the change.
Alternatively, migrate.baseCptCacheSize can be set for powered on virtual machines using PowerCLI. The following commands would set the VMX option without downtime. When migrating the virtual machine, the new VMX process is created on the target host and the buffer is set to the size defined with the option, so no vMotion failure occurs:
$vms = Get-VM (define your list of virtual machines here) | Get-View $vmx = New-Object VMware.Vim.VirtualMachineConfigSpec $vmx.extraConfig += New-Object VMware.Vim.OptionValue $vmx.extraConfig[0].key = "migrate.baseCptCacheSize" $vmx.extraConfig[0].value = "16777216" ($vms).ReconfigVM_Task($vmx)
Note: You might need to increase the value for VMs with larger resolutions, do so at least in 2 MB increments (MB x 1024 x 1024). The overhead of this increase will be limited to the duration of the vMotion.
Additional Information
Feedback
