Snapshot quiescing fails on Linux guests after upgrading to VMware vSphere ESXi and VMware Tools 5.1 and 5.5
search cancel

Snapshot quiescing fails on Linux guests after upgrading to VMware vSphere ESXi and VMware Tools 5.1 and 5.5

book

Article ID: 342815

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

  • Linux guest virtual machines fail to take quiesced snapshots after upgrading ESXi and VMware Tools to 5.1 and 5.5.
  • Using the vSphere Client to take a quiesced snapshot fails immediately with the error:

    The guest OS has reported an error during quiescing. The error code was: 3 The error message was: Error when enabling the sync provider.

  • The hostd.log file, located at /var/log/ of the host, running the virtual machine contains entries similar to:

    [3CB9EB90 verbose 'vm:/vmfs/volumes/<datastore>/<VMNAME>/<VMNAME>.vmx'] Handling message _vmx##: The guestOS has reported an error during quiescing.
    --> The error code was: 3
    --> The error message was: Error when enabling the sync provider.


  • In a Red Hat Enterprise Linux (RHEL) virtual machine, the messages.log file located at /var/log/ contains entries similar to:

    [ debug] [vmbackup] *** VmBackupStart
    [ debug] [vmbackup] Using quiesceApps = 1, quiesceFS = 1, allowHWProvider = 1,execScripts = 1, scriptArg = , timeout = 0
    [ debug] [vmbackup] Quiescing volumes: 6000c296-####-####-####-########021
    [ debug] [vmbackup] *** VmBackup_SendEvent
    [ debug] [vmbackup] *** VmBackupStartScripts
    [ debug] [vmbackup] Trying to run scripts from /etc/vmware-tools/backupScripts.d
    [ debug] [vmbackup] *** VmBackupAsyncCallback
    [ debug] [vmbackup] VmBackupAsyncCallback: checking VmBackupOnFreeze
    [ debug] [vmbackup] Async request 'VmBackupOnFreeze' completed
    [ debug] [vmbackup] *** VmBackupEnableSync
    [ info] [guestinfo] Poll loop disabled.
    [ debug] [vmbackup] *** VmBackupSyncDriverStart
    [ debug] [vmsvc] SyncDriver: Freezing using Linux ioctls...
    [ debug] [vmsvc] SyncDriver: failed to open '########-####-####-####-#########': 2 (No such file or directory)
    [ warning] [vmbackup] Error freezing filesystems

  • With VMwaretools debug logging enabled, you may see messages similar to:

    <YYYY-MM-DD><time>| vcpu-0| I120: Guest: [ debug] [vmsvc:vmsvc] SyncDriver: Freezing using Linux ioctls...</time>

    Or

    <YYYY-MM-DD><time>| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
    <YYYY-MM-DD><time>| vmx| I120: Tools: Tools heartbeat timeout.
    <YYYY-MM-DD><time>| vmx| I120: GuestRpcSendTimedOut: message to toolbox timed out.
    <YYYY-MM-DD><time>| vmx| I120: GuestRpc: app toolbox's second ping timeout; assuming app is down
    <YYYY-MM-DD><time>| vmx| I120: ToolsBackup: changing quiesce state: STARTED -> DONE
    <YYYY-MM-DD><time>| vmx| I120: SnapshotVMXTakeSnapshotComplete: done with snapshot 'smvi_ee2e62bb-####-####-####-########702': 0
    <YYYY-MM-DD><time>| vmx| I120: SnapshotVMXTakeSnapshotComplete: Snapshot 0 failed: Failed to quiesce the virtual machine (40).

    </time></time></time></time></time></time></time>
  • For more information, see Enabling debug logging for VMware Tools within a guest operating system (1007873)

Environment

  • VMware vCenter Server 5.1.x
  • VMware vSphere ESXi 5.1
  • VMware vCenter Server 5.5.x
  • VMware vSphere ESXi 5.5

Cause

There can be two possible causes for this issue:

Note: Red Hat has addressed this issue in bug 768304. You need Red Hat customer login credentials to view the bug details.

Resolution

  • You need to check and possibly update the kernel of your Linux guest virtual machine. If your Linux is RHEL, contact Red Hat for clarification about the release which can reliably support the FIFREEZE/FITHAW ioctl feature. For other vendor's Linux releases, you may have to contact your vendor to determine the Linux kernel used. For more information, see the Additional Information section.

  • Alternatively, you can downgrade from ESXi 5.1 or 5.5 tools to ESXi 5.0 tools on the guest OS. This will force the use of the VMware vmsync driver to do the quiesce snapshot and bypass the issue.

Additional Information