Snapshot quiescing fails on Linux guests

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Linux guest virtual machines fail to take quiesced snapshots
• Using the vSphere Client to take a quiesced snapshot fails immediately with the error:
  
  The guest OS has reported an error during quiescing. The error code was: 3 The error message was: Error when enabling the sync provider.
  
  
• The hostd.log file, located at /var/log/ of the host, running the virtual machine contains entries similar to:
  
  [3CB9EB90 verbose 'vm:/vmfs/volumes/<datastore>/<VMNAME>/<VMNAME>.vmx'] Handling message _vmx##: The guestOS has reported an error during quiescing.
--> The error code was: 3
--> The error message was: Error when enabling the sync provider.
  
  
• In a Red Hat Enterprise Linux (RHEL) virtual machine, the messages.log file located at /var/log/ contains entries similar to:
  
  [ debug] [vmbackup] *** VmBackupStart
  [ debug] [vmbackup] Using quiesceApps = 1, quiesceFS = 1, allowHWProvider = 1,execScripts = 1, scriptArg = , timeout = 0
  [ debug] [vmbackup] Quiescing volumes: 6000c296-####-####-####-########021
  [ debug] [vmbackup] *** VmBackup_SendEvent
  [ debug] [vmbackup] *** VmBackupStartScripts
  [ debug] [vmbackup] Trying to run scripts from /etc/vmware-tools/backupScripts.d
  [ debug] [vmbackup] *** VmBackupAsyncCallback
  [ debug] [vmbackup] VmBackupAsyncCallback: checking VmBackupOnFreeze
  [ debug] [vmbackup] Async request 'VmBackupOnFreeze' completed
  [ debug] [vmbackup] *** VmBackupEnableSync
  [ info] [guestinfo] Poll loop disabled.
  [ debug] [vmbackup] *** VmBackupSyncDriverStart
  [ debug] [vmsvc] SyncDriver: Freezing using Linux ioctls...
  [ debug] [vmsvc] SyncDriver: failed to open '########-####-####-####-#########': 2 (No such file or directory)
  [ warning] [vmbackup] Error freezing filesystems
  
  
• With VMwaretools debug logging enabled, you may see messages similar to:
  
  <YYYY-MM-DD><time>| vcpu-0| I120: Guest: [ debug] [vmsvc:vmsvc] SyncDriver: Freezing using Linux ioctls...</time>
  
  Or
  
  <YYYY-MM-DD><time>| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
  <YYYY-MM-DD><time>| vmx| I120: Tools: Tools heartbeat timeout.
  <YYYY-MM-DD><time>| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
  <YYYY-MM-DD><time>| vmx| I120: GuestRpc: app toolbox's second ping timeout; assuming app is down
  <YYYY-MM-DD><time>| vmx| I120: ToolsBackup: changing quiesce state: STARTED -> DONE
  <YYYY-MM-DD><time>| vmx| I120: SnapshotVMXTakeSnapshotComplete: done with snapshot 'smvi_ee2e62bb-####-####-####-########702': 0
  <YYYY-MM-DD><time>| vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40).
  
  </time></time></time></time></time></time></time>
• For more information, see Enabling debug logging for VMware Tools within a guest operating system (1007873)

Environment

VMware vSphere ESXi

Cause

There can be two possible causes for this issue:

• If the disk.EnableUUID in the virtual_machine.vmx is set to true this causes the incorrect path to be supplied to the function that quiesces the file system in question. For more information, see Snapshot quiescing fails on Linux guests when disk.enableUUID is set to true in the vmx file (2079220)
  
  
• The FIFREEZE/FITHAW ioctl feature within the guest is utilized to quiesce the Linux filesystem. This issue is at the Linux kernel level and would need to be investigated by the vendor.

Resolution

• You need to check and possibly update the kernel of your Linux guest virtual machine. If your Linux is RHEL, contact Red Hat for clarification about the release which can reliably support the FIFREEZE/FITHAW ioctl feature.
  For other vendor's Linux releases, you may have to contact your vendor to determine the Linux kernel used. 

Additional Information

• Enabling debug logging for VMware Tools within a guest operating system
• Snapshot quiescing fails in Linux guests when disk.enableUUID is set to true in the vmx file