Taking a snapshot of a virtual machine after a Storage vMotion on an ESX/ESXi 4.1 or 5.0 host fails
search cancel

Taking a snapshot of a virtual machine after a Storage vMotion on an ESX/ESXi 4.1 or 5.0 host fails

book

Article ID: 344265

calendar_today

Updated On:

Products

VMware vCenter Server VMware vSphere ESXi

Issue/Introduction

Symptoms:
After performing a Storage vMotion of a Windows 2008 or 2008 R2 Virtual machine on an ESX/ESXi 4.1u1 and 5.0 host you experience the following:
  • Attempting to backup or snapshot the Virtual machine fails with the error:

    Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine

  • Taking a manual snapshot is successful when including both snapshot the Virtual machine's memory and quiesce guest file system, but fails when only quiesce guest file system is selected.

  • The hostd.log file may show errors similar to:

    F5909B90 error 'vm:/vmfs/volumes/datatore/virtual_machine/virtual_machine.vmx' opID=2c7d0baf-f9] Quiesced snapshot backup agent error: Asynchronous operation failed: VssSyncStart (5)
    message = "Asynchronous operation failed: VssSyncStart",
    F5548B90 error 'vm:/vmfs/volumes    /datatore/virtual_machine/virtual_machine.vmx' opID=83a9f34b-8a] Quiesced snapshot backup agent error: Asynchronous operation failed: VssSyncStart (5)
    message = "Asynchronous operation failed: VssSyncStart",

  • The vmware.log file may show errors similar to:

    DISKLIB-VMFS : "/vmfs/volumes/datatore/virtual_machine/virtual_machine-flat.vmdk" : open successful (21) size = 53687091200, hd = 0. Type 3
    DISKLIB-VMFS : "/vmfs/volumes    /datatore/virtual_machine/virtual_machine-flat.vmdk" : closed.
    DISKLIB-VMFS : "/vmfs/volumes    /datatore/virtual_machine/virtual_machine-flat.vmdk" : open successful (21) size = 64424509440, hd = 0. Type 3
    DISKLIB-VMFS : "/vmfs/volumes    /datatore/virtual_machine/virtual_machine-flat.vmdk" : closed.
    SOCKET 1 (99) recv detected client closed connection
    Vix: [202112 mainDispatch.c:2472]: VMAutomation: Connection Error (4) on connection 0.
    VmdbPipeStreamsOvlError Couldn't read: OVL_STATUS_EOF

  • If you are enable VMware Tools in debug mode, the vmware.log file of the virtual machine reports errors similar to:

    [10572] [debug] [vmbackup] *** CVmVssRequestor::QueryStatus
    [8092] SnapRequestor: CVmSnapshotRequestor::QueryStatus
    [8092] VssProvider_IsDiskSupported: vendor id is null.
    [8092] VDSHelper::ForEachVolume: return (0)
    [8092] CVmSnapshotRequestor::AddSupportedVolumes: return (8004230c) <---- search for this number
    [8092] CVmSnapshotRequestor::PrepareBackup():1080: failed call: ret = AddSupportedVolumes(), result = 8004230c
    [8092] SnapRequestor: CVmSnapshotRequestor::Finalize
    [8092] SnapRequestor: CVmSnapshotRequestor::UnregisterProvider
    [8092] SnapRequestor: Provider unregistered.
    [10572] [ debug] [vmbackup] *** VmBackupAsyncCallback
    [10572] [ debug] [vmbackup] VmBackupAsyncCallback: checking VssSyncStart
    [10572] [ debug] [vmbackup] *** VmVssBackupOpQuery
    [10572] [ debug] [vmbackup] *** CVmVssRequestor::QueryStatus
    [8092] SnapRequestor: CVmSnapshotRequestor::QueryStatus
    [10572] [ debug] [vmbackup] Async operation error: 0x8004230c
    [10572] [ debug] [vmbackup] *** VmBackup_SendEvent
    [10572] [ debug] [vmbackup] *** VmVssBackupOpRelease

    For more information on enabling VMware Tools in debug mode, see Enabling debug logging for VMware Tools within a guest operating system (1007873).

  • When checking the virtual machine disk properties using PowerShell you see that the Serial Number for the disk is blank:

    C:\> Get-WmiObject -ComputerName <VM_hostname> Win32_DiskDrive -credential <domain>\<user> | Format-List -property *

    The serial number value is missing from the output:

    VM with disk.enableUUID = "TRUE"
    SCSITargetId : 0
    SerialNumber : <-------- The disk UUID/Serial number is blank
    Signature : 357258723
  • In the Application Event logs of the Event Viewer within the Windows guest operating system, you see the error:

    "Volume Shadow Copy Service error: Unexpected error DeviceIoControl"

    Operation:
    Exposing Recovered Volumes
    Locating shadow-copy LUNs
    PostSnapshot Event
    Executing Asynchronous Operation

    Context:
    Device: \\?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
    Examining Detected Volume: Existing - \\?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}
    Execution Context: Provider
    Provider Name: VMware Snapshot Provider
    Provider Version: 1.0.0
    Provider ID: {564d7761-7265-2056-5353-2050726f7669}
    Current State: DoSnapshotSet

  • The Tasks & Events tab of the vSphere Client reports the error:

    'VssSyncStart' operation failed: IDispatch error #8460 (0x8004230c)


Environment

VMware ESX 4.1.x
VMware vCenter Server 5.0.x
VMware vSphere ESXi 5.0
VMware vCenter Server 4.1.x
VMware ESXi 4.1.x Embedded
VMware ESXi 4.1.x Installable

Cause

This issue is caused when the Microsoft VSS service is attempting to do an application quiesced snapshot but is unable to read the disk UUID or Serial Number value for one or more virtual machine disks.
For more information on application quiesced snapshots, see Enabling Windows 2008 application-consistent quiescing on ESX/ESXi 4.1 (1028881).

With ESX/ESXi 4.1 Update 1 and later, support for application quiesced snapshots for Windows 2008 and 2008 R2 Virtual machines was introduced. Application quiesced snapshot support required the new disk.enableUUID parameter to be added to the Virtual machine's .vmx configuration file to enable or disable the feature. When the parameter has been enabled, you need to reboot the virtual machine to ensure that the disk UUID/Serial Number is also populated as it is required for application snapshots.

The disk.enableUUID parameter did not exist on earlier versions of ESX/ESXi, therefore virtual machines created using those earlier versions will not have that parameter included in their .vmx configuration file. If the ESX/ESXi hosts are upgraded from an earlier version to ESX/ESXi 4.1, support for application quiesced snapshots will be available.

When a Virtual machine is migrated to a new datastore using Storage vMotion, the disk.enableUUID parameter is added to the Virtual machine's .vmx configuration file with a value of true if the parameter is not already present in the file. Before the migration, the effective value of disk.enableUUID is false (as the feature is not being used), thus the migration is changing the default value from false to true.

When a backup or snapshot attempts to quiesce the Virtual machine, it will see that application quiescing has been enabled (disk.enableUUID=true), but it will be unable to read the disk UUID/Serial Number as the virtual machine has not been rebooted since that parameter was added to the .vmx configuration file.

Resolution

This issue is resolved in ESX/ESXi 4.1 Update 3, ESXi 5.0 Update 2 and ESXi 5.1 Update 1. For further details see:
ESXi 4.1 U3 Release Notes - Resolved Issues
ESXi 5.0 U2 Release Notes - Resolved Issues
ESXi 5.1 U1 Release Notes - Resolved Issues

To work around this issue in earlier builds of ESX/ESXi, use one of these options:
  • Allow VSS application quiescing to work by generating Disk UUID/Serial number. This can be achieved by leaving the disk.enableUUID value to true and power cycle of the virtual machine.
  • Disable VSS application quiescing feature by setting one of these options.

    Note: This is a non-disruptive process and does not require Virtual machine downtime.
    • Disable application quiescing by disabling the option from VMware Tools.
      1. In the Windows guest, create the file %SystemDrive%\ProgramData\VMware\VMware Tools\tools.conf.
      2. Add these entries to the tools.conf file:

        [vmbackup]
        vss.disableAppQuiescing = true
      3. Restart the VMware Tools Service.

    • Change the value of the disk.enableUUID parameter from true to false by modifying the virtual machine's configuration file (vmname.vmx) and migrate the virtual machine to another host in the environment to reload the virtual machine configuration into the host memory.

      Note: You may use PowerCLI scripts to modify the value on multiple virtual machines simultaneously. VMware Support does not provide PowerCLI scripts under standard support agreement and, therefore, you may have to refer the PowerCLI user guide for creating your own script or contact VMware SDK support.


Additional Information



Enabling debug logging for VMware Tools within a guest operating system
Enabling and disabling Windows 2008 and later application-consistent quiescing on ESXi/ESX
ESX/ESXi 4.1 または 5.0 ホスト上で Storage vMotion が失敗した後に仮想マシンのスナップショットを取得する
在 ESX/ESXi 4.1 或 5.0 主机上执行 Storage vMotion 后创建虚拟机快照失败

Powered by Wolken Software