Snapshot quiescing fails on Linux guests after upgrading to VMware vSphere ESXi and VMware Tools 5.1 and 5.5

Products

• VMware vCenter Server
• VMware vSphere ESXi

Issue/Introduction

• Linux guest virtual machines fail to take quiesced snapshots after upgrading ESXi and VMware Tools to 5.1 and 5.5.
• Using the vSphere Client to take a quiesced snapshot fails immediately with the error:
  
  The guest OS has reported an error during quiescing. The error code was: 3 The error message was: Error when enabling the sync provider.
  
  
• The hostd.log file, located at /var/log/ of the host, running the virtual machine contains entries similar to:
  
  [3CB9EB90 verbose 'vm:/vmfs/volumes/<datastore>/<VMNAME>/<VMNAME>.vmx'] Handling message _vmx##: The guestOS has reported an error during quiescing.
--> The error code was: 3
--> The error message was: Error when enabling the sync provider.
  
  
• In a Red Hat Enterprise Linux (RHEL) virtual machine, the messages.log file located at /var/log/ contains entries similar to:
  
  [ debug] [vmbackup] *** VmBackupStart
  [ debug] [vmbackup] Using quiesceApps = 1, quiesceFS = 1, allowHWProvider = 1,execScripts = 1, scriptArg = , timeout = 0
  [ debug] [vmbackup] Quiescing volumes: 6000c296-####-####-####-########021
  [ debug] [vmbackup] *** VmBackup_SendEvent
  [ debug] [vmbackup] *** VmBackupStartScripts
  [ debug] [vmbackup] Trying to run scripts from /etc/vmware-tools/backupScripts.d
  [ debug] [vmbackup] *** VmBackupAsyncCallback
  [ debug] [vmbackup] VmBackupAsyncCallback: checking VmBackupOnFreeze
  [ debug] [vmbackup] Async request 'VmBackupOnFreeze' completed
  [ debug] [vmbackup] *** VmBackupEnableSync
  [ info] [guestinfo] Poll loop disabled.
  [ debug] [vmbackup] *** VmBackupSyncDriverStart
  [ debug] [vmsvc] SyncDriver: Freezing using Linux ioctls...
  [ debug] [vmsvc] SyncDriver: failed to open '########-####-####-####-#########': 2 (No such file or directory)
  [ warning] [vmbackup] Error freezing filesystems
  
  
• With VMwaretools debug logging enabled, you may see messages similar to:
  
  <YYYY-MM-DD><time>| vcpu-0| I120: Guest: [ debug] [vmsvc:vmsvc] SyncDriver: Freezing using Linux ioctls...</time>
  
  Or
  
  <YYYY-MM-DD><time>| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
  <YYYY-MM-DD><time>| vmx| I120: Tools: Tools heartbeat timeout.
  <YYYY-MM-DD><time>| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
  <YYYY-MM-DD><time>| vmx| I120: GuestRpc: app toolbox's second ping timeout; assuming app is down
  <YYYY-MM-DD><time>| vmx| I120: ToolsBackup: changing quiesce state: STARTED -> DONE
  <YYYY-MM-DD><time>| vmx| I120: SnapshotVMXTakeSnapshotComplete: done with snapshot 'smvi_ee2e62bb-####-####-####-########702': 0
  <YYYY-MM-DD><time>| vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40).
  
  </time></time></time></time></time></time></time>
• For more information, see Enabling debug logging for VMware Tools within a guest operating system (1007873)

Environment

• VMware vCenter Server 5.1.x
• VMware vSphere ESXi 5.1
• VMware vCenter Server 5.5.x
• VMware vSphere ESXi 5.5

Cause

There can be two possible causes for this issue:

• If the disk.EnableUUID in the virtual_machine.vmx is set to true this causes the incorrect path to be supplied to the function that quiesces the file system in question. For more information, see Snapshot quiescing fails on Linux guests when disk.enableUUID is set to true in the vmx file (2079220)
  
  
• The FIFREEZE/FITHAW ioctl feature within the guest is utilized to quiesce the Linux filesystem. This issue is at the Linux kernel level and would need to be investigated by the vendor.

Note: Red Hat has addressed this issue in bug 768304. You need Red Hat customer login credentials to view the bug details.

Resolution

• You need to check and possibly update the kernel of your Linux guest virtual machine. If your Linux is RHEL, contact Red Hat for clarification about the release which can reliably support the FIFREEZE/FITHAW ioctl feature. For other vendor's Linux releases, you may have to contact your vendor to determine the Linux kernel used. For more information, see the Additional Information section.
  
  
• Alternatively, you can downgrade from ESXi 5.1 or 5.5 tools to ESXi 5.0 tools on the guest OS. This will force the use of the VMware vmsync driver to do the quiesce snapshot and bypass the issue.

Additional Information

• Some vendors may not have a Linux release from kernel version 2.6.35-22. For instance, Ubuntu does, and RHEL does not. If your vendor does not have this particular or a later 2.6.35 release available, contact them for clarification. To see the Linux kernel versions associated with each RedHat version, see the document Red Hat Enterprise Linux Release Dates documentation.
• For more information, see the Linux kernel releases section in the Linux Kernel 2.6 Release notes.
• Enabling debug logging for VMware Tools within a guest operating system
• Snapshot quiescing fails in Linux guests when disk.enableUUID is set to true in the vmx file