Linux VMs under heavy I/O load may hang during snapshot quiescing

Products

VMware vSphere ESXi

Issue/Introduction

Virtual machines (VMs) running Linux as guest operating system (guest OS) might become unresponsive when attempting quiesced snapshot operation while under heavy I/O load.

With VMware Tools debug logging enabled, the /var/log/vmware-vmsvc-root.#.log inside the guest OS contains messages similar to:

<timestamp> [ warning] [vmbackup] [1154] Failed to send vmbackup event: vmbackup.eventSet req.keepAlive 0 , result: Unknown command.
<timestamp> [ warning] [vmbackup] [1154] Canceling backup operation due to timeout.
<timestamp> [ warning] [vmbackup] [1154] Failed to send vmbackup event: vmbackup.eventSet req.aborted 4 Quiesce canceled., result: Unknown command.
<timestamp> [ warning] [vmbackup] [1154] Failed to send vmbackup event: vmbackup.eventSet req.done 0 , result: Unknown command.
<timestamp> [ warning] [vmsvc] [3546888] SyncDriver: '/db-data' appears locked or frozen by another process.  Cannot complete the quiesced snapshot request.
<timestamp> [ warning] [vmbackup] [3546888] Error trying to perform OP_FREEZE on filesystems.
<timestamp> [ warning] [vmsvc] [3546889] SyncDriver: '/db-data' appears locked or frozen by another process.  Cannot complete the quiesced snapshot request.
<timestamp> [ warning] [vmbackup] [3546889] Error trying to perform OP_FREEZE on filesystems.
<timestamp> [ warning] [vmsvc] [3546891] SyncDriver: '/db-data' appears locked or frozen by another process.  Cannot complete the quiesced snapshot request.
<timestamp> [ warning] [guestinfo] [1154] *** WARNING: GuestInfo collection interval longer than expected; actual=1109 sec, expected=30 sec. ***

A SysRq-c panic might be triggered by the Oracle cssdmonitor due to a filesystem freeze associated with the snapshot operation:
```
Kernel panic - not syncing: sysrq triggered crash
20 of 21 xfs filesystems are frozen
```

The log of the virtual machine process, /vmfs/volumes/<datastore>/<vm_name>/vmware.log contains the following entries:

<timestamp> In(05) vmx 5127ee63-59-91d2 SNAPSHOT: SnapshotPrepareTakeDoneCB: Prepare phase complete (The operation completed successfully).
<timestamp> In(05) vcpu-5 - ToolsBackup: changing quiesce state: IDLE -> STARTED
<timestamp> In(05) vmx 5127ee63-59-91d2 Msg_Post: Warning
<timestamp> In(05) vmx 5127ee63-59-91d2 [msg.snapshot.quiesce.timeout] Timed out while quiescing the virtual machine.
<timestamp> In(05) vmx 5127ee63-59-91d2 ----------------------------------------
<timestamp> In(05) vmx - ToolsBackup: changing quiesce state: STARTED -> DONE

Environment

VMware vSphere ESXi 7.0.x
VMware vSphere ESXi 8.0.x
Open-VM-Tools

Cause

In order to create or consolidate a quiesced snapshot, the VMware Tools vmbackup plugin will attempt to freeze the guest filesystem.
However, when the VM is under heavy I/O load, a condition might occur that results in the vmtools freeze operation failing and instead timing out after the default timeout of 15 minutes, despite the filesystem being frozen in actuality. Because of this condition, further writes to the file system might be attempted, potentially resulting in both vmtools and the VM itself becoming unresponsive.

Resolution

VMware engineering is aware of this issue and actively working towards implementing a fix in a future VMware Tools version. Subscribe to this article to receive an email when this article is updated.
In the meantime consider mitigating the issue by using either of the following workarounds:

Use pre-freeze/post-thaw scripts to temporarily stop any workloads causing heavy I/O load

Within the guest operating system create a script /etc/vmware-tools/backupScripts.d/pre-freeze-post-thaw-linux.sh
Note: If the folder does not exist, create it. The script must be located in this specific location for it to be triggered during quiescing operations.

Edit the file and populate it with the following content:

#!/bin/sh
# Copyright 2022 VMware, Inc. All rights reserved. -- VMware Confidential
# Description: Pre-Freeze / Post-Thaw script for quiesced backups.
#
# Copy to /etc/vmware-tools/backupScripts.d/ on the guest.
# During quiesced backup all scripts in this directory are called by VM tools.
# Script is invoked with a single parameter freeze|thaw|freezeFail.
# This script just logs invocation time into /var/log/freeze.log
# Log time is UTC, example of timestamp: 2022-10-26T17:55:13Z

log="/var/log/freeze.log"
today=`date -u +%Y-%m-%dT%H:%M:%SZ`
if [[ $1 == "freeze" ]]
then
  echo "${today}: This is quiescing freeze script" >> ${log}
elif [[ $1 == "thaw" ]]
then
  echo "${today}: This is quiescing thaw script" >> ${log}
elif [[ $1 == "freezeFail" ]]
then
  echo "${today}: freezeFail - Quiescing failed" >> ${log}
else
  echo "No argument was provided"
fi

If debug logging is enabled for VMware Tools, ensure that it is configured to write into the VMX log outside of the guest OS, using the <plugin_name>.handler = vmx setting in the /etc/vmware-tool/tools.conf file. For example:
```
[logging]
log = true
vmtoolsd.level = debug
vmtoolsd.handler = vmx
vmsvc.level = debug
vmsvc.handler = vmx
```
Note: Logging might be configured for other components, such as vmusr, vmvss or vmbackup etc. Make sure that the correct handler is configured for every one of these components. Refer to Enabling debug logging for VMware Tools within a guest operating system for more information.

Consider using alternative backup methods
Using pre-freeze/post-thaw scripts might not always be an option, depending on the specific workload running within the virtual machine(s). If this is the case, consider using alternative means to backup, for example by using an agent-based method to backup the data inside the virtual machine rather than working image-based.
Schedule backups outside of peak hours
If possible, consider when the specific virtual machine might be under heavy I/O and when not. If it is any possible, move the backup schedule for the VM, so backups are taken at times when I/O is least high to prevent the conditions leading to the failure from occurring in the first place.