A virtual machine can freeze under load when you take quiesced snapshots or use custom quiescing scripts

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• VMware products require file systems within a guest operating system to be quiesced prior to a snapshot operation for the purposes of backup and data integrity.

   

• Virtual machine generating heavy I/O workload may encounter issues when quiescing prior to a snapshot operation. These issues may be related to the component that does the quiescing or the custom quiescing scripts as described in the Virtual Machine Backup Guide.

   

• Services which have been known to generate heavy I/O workload include, but are not limited to, Exchange, Active Directory, LDAP, and databases.

   

• The quiescing operation is done by an optional VMware Tools component called the SYNC driver.

• Quiescing task is also performed by Microsoft's Volume Shadow Copy Service (VSS). VSS is provided by Microsoft in their operating systems as of Windows Server 2003 and Windows XP.

• Operating systems which do not have the Volume Shadow Copy Service make use of the SYNC driver for quiescing operations.

• Virtual machine could also freeze when the I/O in the virtual machine is high and the quiescing operation is unable to flush all the data to disk, while further I/O is created.

• This issue also occurs if you try to create a quiesced snapshot on a virtual machine that does not have free space on the underlying  virtual machine hard disk.
  
  For more information, see Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

   

   

Environment

VMware vSphere ESXi 8.x
VMware vSphere ESXi 7.x
VMware vSphere ESXi 6.x
VMware vSphere ESXi 5.x

Resolution

If the guest operating system is Windows Vista, Windows Server 2003, or other newer Windows operating systems, a Volume Shadow Copy Services (VSS) module is installed.

 

For other, earlier Windows operating systems, the Filesystem Sync driver is installed. These modules allow external third-party back up software that is integrated with vSphere to create application-consistent snapshots. 

 

 

• Quiescing with the SYNC Driver

• • During the snapshot process, certain processes are paused and virtual machine disks are quiesced. The modules also support quiescing snapshot on Linux OS
  • If installed, the SYNC driver holds incoming I/O writes while it flushes all dirty data to a disk, thus making file systems consistent. Under heavy loads, the delay in I/O can become too long, which affects many time-sensitive applications, including the services which generate the heavy I/O (such as an Exchange Server). If writes issued by these services get delayed for too long, the service may stop and issues error messages.
  • To avoid this issue, disable the SYNC driver or stop the service generating heavy I/O before taking a snapshot.

Note: The sync driver is only required for legacy versions of Windows such as Windows XP and Windows 2000 which do not include the Microsoft VSS service. Updated versions of VMware Tools will automatically uninstall the SYNC driver.

 

• Disabling the VCB SYNC Driver (LGTO_Sync)
  
  
  • Disabling the SYNC driver allows you to keep the heavy I/O services on-line, but results in snapshots being only crash-consistent.
    • To disable the VCB SYNC driver:
      1. In the Windows Device Manager, click View > Show hidden devices.
      2. Expand Non-Plug and Play Drivers.
      3. Right-click Sync Driver and click Disable.
      4. Click Yes twice to disable the device and restart the computer.
      5. If the SYNC driver is still enables, see VMware Tools does not remove SYNC driver

 

• Stopping services generating heavy I/O
  • Use the following pre-freeze and post-thaw scripts to take the service generating heavy I/O offline for approximately 60 seconds and then restart it again after the snapshot is taken. This approach leaves the service inactive, but keeps the SYNC driver enabled while the snapshot is taken, ensuring application consistency. Using this method, you create the quiesced snapshot of guest operating system
    
    Note: Make sure of VMware tools is installed.
    
    
  • This example shuts down Exchange Services prior to a quiescing operation:

    C:\Windows\pre-freeze-script.bat
     
    @echo off
    net stop MSExchangeSA /yes
     
    C:\Windows\post-thaw-script.bat
    @echo off
    Net Start MsExchangeSA
    Net Start MsExchangeIS
    Net Start MsExchangeMTA
    

    
    For additional information, see the Virtual Machine Backup Guide.

• Quiescing with the Volume Shadow Copy Service
  • If you experience issues quiescing while using Microsoft's VSS, see Troubleshooting Volume Shadow Copy (VSS) quiesce related issues