Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine
search cancel

Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

book

Article ID: 340150

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

You may experience the following:
  • Creating a quiesced snapshot fails.
  • While taking a snapshot, the following error is seen:

    Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine
     
  • This issue is experienced when doing a hot clone of a virtual machine. The error seen is:

    An error occurred while quiescing the virtual machine. See the virtual machine's event log for details 'VssSyncStart' operation failed: IDispatch error #8449 (0x80042301)
     
  • The vpxa.log file contains entries similar to:

    YYYY-MM-DDTHH:MM:SS.077Z [45EEEB90 info 'Default' opID=D1D8F924-00002337-83] [VpxLRO] -- ERROR task-768 -- -- vpxapi.VpxaService.createSnapshot: vim.fault.ApplicationQuiesceFault:
    --> Result:
    (vim.fault.ApplicationQuiesceFault) {
    dynamicType = unset,
    faultCause = (vmodl.MethodFault) null,
    faultMessage = (vmodl.LocalizableMessage) [
    (vmodl.LocalizableMessage) {
    dynamicType = unset,
    key = "msg.snapshot.quiesce.vmerr",
    arg = (vmodl.KeyAnyValue) [
    (vmodl.KeyAnyValue) {
    dynamicType = unset,
    key = "1",
    value = "5",
    },
    (vmodl.KeyAnyValue) {
    dynamicType = unset,
    key = "2",
    value = "'VssSyncStart' operation failed: IDispatch error #8449 (0x80042301)",
    }
    ],
    message = "The Guest OS has reported an error during quiescing.
    --> The error code was: 5
    --> The error message was: 'VssSyncStart' operation failed: IDispatch error #8449 (0x80042301)
    --> ",
    }
    ],
    msg = "An error occurred while quiescing the virtual machine. See the virtual machine's event log for details."
    }
    --> Args:
    --> Arg vmid:6
    --> Arg name:"clone-temp-#########"
    --> Arg description:
    "This temporary snapshot is taken as part of the clone operation. The temporary snapshot will be deleted once the clone operation is complete."
    --> Arg memory:false
    --> Arg quiesce:true
     
  • The vmware.log file of the virtual machine contains entries similar to:

    YYYY-MM-DDTHH:MM:SS.332Z| vmx| I120: Msg_Post: Warning
    YYYY-MM-DDTHH:MM:SS.333Z| vmx| I120: [msg.snapshot.quiesce.vmerr] The guest OS has reported an error during quiescing.
    YYYY-MM-DDTHH:MM:SS.333Z| vmx| I120+ The error code was: 5
    YYYY-MM-DDTHH:MM:SS.333Z| vmx| I120+ The error message was: 'VssSyncStart' operation failed: IDispatch error #8451 (0x80042303)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on the environment.

Environment

VMware vCenter Server 7.x
VMware vCenter Server 8.x
VMware ESXi 7.x
VMware ESXi 8.x

Cause

This issue occurs when the I/O in the virtual machine is high and the quiescing operation is unable to flush all the data to disk, while further I/O is created.

This issue also occurs when trying to create a quiesced snapshot on a virtual machine that does not have free space. That is, if the free space on the virtual machine is 0 bytes.

Resolution

Note: It is reported that this issue occurs after installing a third-party backup agent inside the virtual machine without rebooting it. If this is the case, try a reboot of the virtual machine to determine if that resolves the issue before engaging in more advanced troubleshooting.

To resolve this issue, perform one of these options:
  • Verify VSS, guest, and backup product configuration. For additional steps on troubleshooting VSS, see Troubleshooting Volume Shadow Copy (VSS) quiesce related issues.
     
  • Reduce the amount of ongoing I/O to the virtual machine. This is accomplished using pre-freeze scripts or post-freeze scripts to quiesce application I/O. 

  • Opt for a crash-consistent snapshot (as opposed to application-consistent) of the virtual machine by avoiding quiescing of the file system.
If quiescing the virtual machine during snapshot creation isn't needed, there are several options. Choose one of these options, based on the environment:
  • If the snapshots are taken through the ESXi host using the vim-cmd command, set the option of quiescing to 0.

    To get a list of virtual machines and their VMID, run the command:

    vim-cmd vmsvc/getallvms

    To set the quiescing to 0, run the command:

    vim-cmd vmsvc/snapshot.create vmid [snapshotName] [snapshotDescription] [includeMemory] [quiesced]

    For example, if a virtual machine named testVM has a VMID of 23, the command is:

    vim-cmd vmsvc/snapshot.create 23 "SnapName" "Snap Description" 0 0
     
  • If the snapshots are taken through the ESXi host terminal using the vmware-cmd command, set the option of quiescing to 0.

    Run this command to set quiescing to 0:

    vmware-cmd cfg createsnapshot name description quiesce memory

    For example:

    vmware-cmd /vmfs/volumes/<datastore-uuid>/VMname/VMname.vmx createsnapshot "Snap Name" "Snap Description" 0 0
     
  • If a third-party backup product is used that does not allow for configuration of non-quiesced snapshots, remove the VSS component from the Windows guest operating system, provided as part of VMware Tools. This results in VMware Tools attempting to use an older sync driver. When a quiesced snapshot is requested, VMware Tools does not find and utilize the VSS driver and then attempts to quiesce using the sync driver. To reinstall VMware Tools without the VSS option:
     
    1. Uninstall VMware Tools (removing VMware Tools removes Vmware Tools features as well as  the VMXNET driver, see Choosing a network adapter for a virtual machine).
    2. Allow the system to reboot.
    3. Reinstall VMware Tools. Ensure to click Custom Install.
    4. Deselect VSS.

    Notes:
    • This requires a reboot of the virtual machine. VMware recommends scheduling downtime before performing this action.
    • The option to deselect VSS and use the sync driver on Windows Server 2003 and later versions of Windows is no longer available. Any snapshot taken with the quiesce option uses VSS to quiesce the guest OS file system by default unless the Volume Shadow Copy service is disabled in the guest OS.
    • After the installation, a snapshot where the quiescing operation is not performed will happen, even if specifically requested.
Notes:
  • Microsoft does not support snapshots of Exchange servers. For more information on support for snapshots.
  • When using the Symantec NetBackup tool with virtual machines running Microsoft Exchange, consider using the Symantec VSS provider. For more information, see the Veritas Netbackup for VMware Administrator's Guide. If VMware Tools was installed before the Symantec VSS, follow the NetBackup Admin Guide for information on uninstalling and reinstalling VMware Tools without the VMWare VSS. If removal of the Symantec VSS is needed, contact Symantec for assistance with uninstalling/reinstalling the Symantec VSS service.
  • The issue occurs if any of the VSS writers are failing. Verify the list of writers before running the backup job and after the job is finished, verify if any VSS writer is failing. As the VSS writers are provided by third-party vendors, engage the vendor for any issues with them or to troubleshoot any failures of those writers.


Additional Information

Powered by Wolken Software