Linux VM Hangs During Quiesced Snapshot Operations with VMware Tools

Products

• CA Privileged Access Manager - Server Control (PAMSC)

Issue/Introduction

Linux virtual machines may become unresponsive or experience a system hang during quiesced snapshot operations. This behavior occurs in environments using VMware Tools, particularly when the virtual machine handles heavy I/O workloads. This article provides steps to mitigate the hang by modifying snapshot scripts.

Environment

• Linux (Red Hat Enterprise Linux 8)
• VMware Tools
• PAMSC (Privileged Access Management Server Control)
• vSphere ESXi

Cause

During a quiesced snapshot, the VMware Tools vmbackup plugin attempts to freeze the filesystem. Under high I/O workloads (e.g., database operations), the freeze operation exceeds the default 15-minute timeout threshold. The filesystem remains in a frozen state, blocking kernel processes and resulting in a system hang.

Resolution

Perform the following steps to configure custom pre-freeze and post-thaw scripts to pause heavy I/O workloads during snapshots:

1. Navigate to the directory /etc/vmware-tools/backupScripts.d/.
2. Create or edit the script pre-freeze-post-thaw-linux.sh.
3. Add commands to stop or pause services that generate heavy I/O (e.g., databases, PAMSC agents, or antivirus) in the pre-freeze section of the script.
4. Add commands to resume these services in the post-thaw section.
5. Ensure the script is executable with chmod +x /etc/vmware-tools/backupScripts.d/pre-freeze-post-thaw-linux.sh.
6. Enable or increase vmtoolsd debug logging to monitor for future occurrences.
7. Monitor snapshot operations to verify the freeze operation completes successfully.

Note: This issue is targeted to be fixed in a future release. 

• For deeper investigation, a core dump is required. Ensure core dumps are enabled in your environment.
• If the issue persists after implementing these workarounds, contact Broadcom Support. To speak with a representative or a Support Engineer, see Contact Support. Scroll to the bottom of the page and click on your respective region.