Quiescing failure on Linux VM with disk.enableUUID set to true

Products

• VMware vSphere ESXi

Issue/Introduction

Symptoms:

• Taking a quiesced snapshot using the vSphere Web Client fails immediately with the error:
  
  The guest OS has reported an error during quiescing. The error code was: 3 The error message was: Error when enabling the sync provider.
   
• Snapshot quiescing fails in Linux guests.
• When disk.enableUUID parameter is set to true in the vmx file, snapshot quiescing fails.
• In the /var/log/hostd.log file, you see the error:
  
  [3CB9EB90 verbose 'vm:/vmfs/volumes/<datastore>/<VMNAME>/<VMNAME>.vmx'] Handling message _vmx##: The guestOS has reported an error during quiescing.
--> The error code was: 3
--> The error message was: Error when enabling the sync provider.
   
• In a Red Hat Enterprise Linux (RHEL) virtual machine, the /var/log/messages.log file contains error similar to:
  
  [ debug] [vmbackup] *** VmBackupStart
  [ debug] [vmbackup] Using quiesceApps = 1, quiesceFS = 1, allowHWProvider = 1,execScripts = 1, scriptArg = , timeout = 0
  [ debug] [vmbackup] Quiescing volumes: 6000c296-####-####-####-########8021
  [ debug] [vmbackup] *** VmBackup_SendEvent
  [ debug] [vmbackup] *** VmBackupStartScripts
  [ debug] [vmbackup] Trying to run scripts from /etc/vmware-tools/backupScripts.d
  [ debug] [vmbackup] *** VmBackupAsyncCallback
  [ debug] [vmbackup] VmBackupAsyncCallback: checking VmBackupOnFreeze
  [ debug] [vmbackup] Async request 'VmBackupOnFreeze' completed
  [ debug] [vmbackup] *** VmBackupEnableSync
  [ info] [guestinfo] Poll loop disabled.
  [ debug] [vmbackup] *** VmBackupSyncDriverStart
  [ debug] [vmsvc] SyncDriver: Freezing using Linux ioctls...
  [ debug] [vmsvc] SyncDriver: failed to open 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx': 2 (No such file or directory)
  [ warning] [vmbackup] Error freezing filesystems
   
• When debug logging is enabled in VMware Tools, you see log entries similar to:
• • <YYYY-MM-DD>T<time>.864Z| vcpu-0| I120: Guest: [ debug] [vmsvc:vmsvc] SyncDriver: Freezing using Linux ioctls...
    
    </time>
  • <YYYY-MM-DD>T<time>.290Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
    <YYYY-MM-DD>T<time>.694Z| vmx| I120: Tools: Tools heartbeat timeout.
    <YYYY-MM-DD>T<time>.290Z| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
    <YYYY-MM-DD>T<time>.290Z| vmx| I120: GuestRpc: app toolbox's second ping timeout; assuming app is down
    <YYYY-MM-DD>T<time>.291Z| vmx| I120: ToolsBackup: changing quiesce state: STARTED -> DONE
    <YYYY-MM-DD>T<time>.291Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: done with snapshot 'vm_########-####-####-####-feb########': 0
    <YYYY-MM-DD>T<time>.291Z| vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40).
    
    </time></time></time></time></time></time></time>For more information, see Enabling debug logging for VMware Tools within a guest operating system (313512).

Environment

VMware vSphere ESXi 5.1
VMware vSphere ESXi 5.5
VMware vSphere ESXi 6.0
VMware vSphere ESXi 6.5

Cause

This issue occurs if the disk.EnableUUID parameter in the virtual_machine.vmx file is set to true. This supplies the incorrect path to the function that quiesces the file system in question.

Resolution

Workaround:
To work around this issue:

• Update the VMware Tools to version 10.1.0 or later.
• Set the disk.EnableUUID parameter to false in the virtual_machine.vmx file.
   
  Note: The disk.EnableUUID parameter is not included in the .vmx file by default. If the parameter does not exist, it is taken as false since this is the default behavior. To modify this setting, it is required to be added to the .vmx file.
   
  To set the disk.EnableUUID parameter to false in the virtual_machine.vmx file:
  1. Connect to the host running the virtual machine with an SSH session.
     
     For more information, see Using ESXi Shell in ESXi 5.x and 6.0 (311213).
      
  2. Navigate to the virtual_machine.vmx file located at /vmfs/volumes/virtual_machine_datastore/ virtual_machine/.
  3. Open the virtual_machine.vmx file using a text editor.
  4. Locate the disk.EnableUUID entry within the file.
  5. Set the disk.EnableUUID entry to false. 
  6. Open a console session to the virtual machine.
  7. Open the tools.conf file, located at the /etc/vmware-tools directory, using a text editor.
     
     Note: If the tools.conf file does not exists, create an empty file by running the vi tools.conf command.
      
  8. Add these parameters to the file:
     
     [vmbackup]
     enableSyncDriver = false
     
     Note: This only runs a sync operation before the snapshot, and does not run a FREEZE on the filesystem.
      
  9. Save and close the file.

Additional Information

Enabling debug logging for VMware Tools within a guest operating system
Using ESXi Shell in ESXi 5.x and 6.x