Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:

Creating a quiesced snapshot fails.
While taking a snapshot, you see the error:

Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine
You experience this issue when you are doing a hot clone of a virtual machine. You see the error:

An error occurred while quiescing the virtual machine. See the virtual machine's event log for details 'VssSyncStart' operation failed: IDispatch error #8449 (0x80042301)
The vpxa.log file contains entries similar to:

YYYY-MM-DDTHH:MM:SS.077Z [45EEEB90 info 'Default' opID=D1D8F924-00002337-83] [VpxLRO] -- ERROR task-768 -- -- vpxapi.VpxaService.createSnapshot: vim.fault.ApplicationQuiesceFault:
--> Result:
(vim.fault.ApplicationQuiesceFault) {
dynamicType = unset,
faultCause = (vmodl.MethodFault) null,
faultMessage = (vmodl.LocalizableMessage) [
(vmodl.LocalizableMessage) {
dynamicType = unset,
key = "msg.snapshot.quiesce.vmerr",
arg = (vmodl.KeyAnyValue) [
(vmodl.KeyAnyValue) {
dynamicType = unset,
key = "1",
value = "5",
},
(vmodl.KeyAnyValue) {
dynamicType = unset,
key = "2",
value = "'VssSyncStart' operation failed: IDispatch error #8449 (0x80042301)",
}
],
message = "The Guest OS has reported an error during quiescing.
--> The error code was: 5
--> The error message was: 'VssSyncStart' operation failed: IDispatch error #8449 (0x80042301)
--> ",
}
],
msg = "An error occurred while quiescing the virtual machine. See the virtual machine's event log for details."
}
--> Args:
--> Arg vmid:6
--> Arg name:"clone-temp-xxxxxxxxx"
--> Arg description:
"This temporary snapshot is taken as part of the clone operation. The temporary snapshot will be deleted once the clone operation is complete."
--> Arg memory:false
--> Arg quiesce:true
The vmware.log file of the virtual machine contains entries similar to:

YYYY-MM-DDTHH:MM:SS.332Z| vmx| I120: Msg_Post: Warning
YYYY-MM-DDTHH:MM:SS.333Z| vmx| I120: [msg.snapshot.quiesce.vmerr] The guest OS has reported an error during quiescing.
YYYY-MM-DDTHH:MM:SS.333Z| vmx| I120+ The error code was: 5
YYYY-MM-DDTHH:MM:SS.333Z| vmx| I120+ The error message was: 'VssSyncStart' operation failed: IDispatch error #8451 (0x80042303)

Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.

Environment

VMware VirtualCenter 2.5.x
VMware ESXi 4.1.x Installable
VMware ESXi 3.5.x Installable
VMware vCenter Server 5.0.x
VMware vCenter Server Appliance 5.1.x
VMware vCenter Server 4.0.x
VMware vCenter Server Appliance 5.0.x
VMware vCenter Server 5.1.x
VMware ESXi 4.0.x Installable
VMware vSphere ESXi 5.1
VMware ESXi 4.0.x Embedded
VMware ESXi 3.5.x Embedded
VMware ESX 4.1.x
VMware vSphere ESXi 5.0
VMware ESX Server 3.5.x
VMware ESX 4.0.x
VMware ESXi 4.1.x Embedded
VMware vCenter Server 4.1.x

Cause

This issue occurs when the I/O in the virtual machine is high and the quiescing operation is unable to flush all the data to disk, while further I/O is created.

This issue also occurs if you try to create a quiesced snapshot on a virtual machine that does not have free space. That is, if the free space on the virtual machine is 0 bytes.

Resolution

Note: It is reported that this issue occurs after installing a third-party backup agent inside the virtual machine without rebooting it. If this is the case, try a reboot of the virtual machine to determine if that resolves the issue before engaging in more advanced troubleshooting.

To resolve this issue, perform one of these options:

Verify VSS, guest, and backup product configuration. For additional steps on troubleshooting VSS, see Troubleshooting Volume Shadow Copy (VSS) quiesce related issues.
Reduce the amount of ongoing I/O to the virtual machine. This is accomplished using pre-freeze scripts or post-freeze scripts to quiesce application I/O. For more information, see the ESXi 6.5 Virtual Machine Administration Guide.
Opt for a crash-consistent snapshot (as opposed to application-consistent) of the virtual machine by avoiding quiescing of the file system.

If you do not want to quiesce the virtual machine during snapshot creation, there are several options. Choose one of these options, based on your environment:

Consider stopping the application before the backup and restarting it using the pre-freeze and post-thaw scripts which are a feature of VMware Tools.

If the snapshots are taken through the ESXi host using the vim-cmd command, set the option of quiescing to 0.

To get a list of virtual machines and their VMID, run the command:

vim-cmd vmsvc/getallvms

To set the quiescing to 0, run the command:

vim-cmd vmsvc/snapshot.create vmid [snapshotName] [snapshotDescription] [includeMemory] [quiesced]

For example, if a virtual machine named testVM has a VMID of 23, the command is:

vim-cmd vmsvc/snapshot.create 23 "SnapName" "Snap Description" 0 0
If the snapshots are taken through the ESXi host terminal using the vmware-cmd command, set the option of quiescing to 0.

Run this command to set quiescing to 0:

vmware-cmd cfg createsnapshot name description quiesce memory

For example:

vmware-cmd /vmfs/volumes/<datastore-uuid>/VMname/VMname.vmx createsnapshot "Snap Name" "Snap Description" 0 0
If you are using a third-party backup product that does not allow you to configure for non-quiescent snapshots, remove the VSS component from the Windows guest operating system, provided as part of VMware Tools. This results in VMware Tools attempting to use an older sync driver. When you request a quiescent snapshot, VMware Tools does not find and utilize the VSS driver and then attempts to quiesce using the sync driver. To reinstall VMware Tools without the VSS option:
1. Uninstall VMware Tools (removing VMware Tools removes Vmware Tools features as well as the VMXNET driver, see Choosing a network adapter for your virtual machine).
2. Allow the system to reboot.
3. Reinstall VMware Tools. Ensure that you click Custom Install.
4. Deselect VSS.
Notes:
- This requires a reboot of the virtual machine. VMware recommends scheduling downtime before performing this action.
- The option to deselect VSS and use the sync driver on Windows Server 2003 and later versions of Windows is no longer available. Any snapshot taken with the quiesce option uses VSS to quiesce the guest OS file system by default unless the Volume Shadow Copy service is disabled in the guest OS.
- After the installation, you are able to take a snapshot where the quiescing operation is not performed, even if specifically requested.

Notes:

Microsoft does not support snapshots of Exchange servers. For more information on support for snapshots.
When using the Symantec NetBackup tool with virtual machines running Microsoft Exchange, consider using the Symantec VSS provider. For more information, see the Veritas Netbackup for VMware Administrator's Guide. If VMware Tools was installed before the Symantec VSS, follow the NetBackup Admin Guide for information on uninstalling and reinstalling VMware Tools without the VMWare VSS. If you need to remove the Symantec VSS, contact Symantec for assistance with uninstalling/reinstalling the Symantec VSS service.
The issue occurs if any of the VSS writers are failing. Verify the list of writers before running the backup job and after the job is finished, verify if any VSS writer is failing. As the VSS writers are provided by third-party vendors, engage the vendor for any issues with them or to troubleshoot any failures of those writers.

Additional Information

For related information, see:

This issue is being checked by Diagnostics for VMware Cloud Foundation.

The check is as follows:

Product: ESXi
Log File: vpxa.log
Log Expression Check "'VssSyncStart' operation failed: IDispatch error #8449"