A virtual machine fails to respond when using backup software and VMware vSphere Replication
search cancel

A virtual machine fails to respond when using backup software and VMware vSphere Replication

book

Article ID: 328490

calendar_today

Updated On:

Products

VMware Live Recovery

Issue/Introduction

Symptoms:
  • During a snapshot consolidation, the virtual machine does not respond.
  • The ESXi host becomes disconnected in vCenter Server.
  • This issue occurs when you attempt to consolidate a snapshot on a virtual machine which has 255 snapshots created by vSphere Replication.

    Note: The maximum supported number of snapshots is 32.

     
  • This issue occurs when vSphere Replication uses Volume Shadow Copy (VSS) to quiesce the snapshots.
  • In vCenter Server, you see the error:

    hierarchy is too deep
     
  • The hostd.log file located at /var/log/ on the ESXi host contains entries similar to:

    T02:22:35.984Z [50B87B90 error 'vm:/vmfs/volumes/4eae978d-########-####-bc305be7c11e/VM/VM.vmx'] Invalid transition requested (VM_STATE_REMOVE_SNAPSHOT -> VM_STATE_CONSOLIDATE_ALL_DISKS): Tasks running

    T02:22:35.984Z [50B87B90 info 'vm:/vmfs/volumes/4eae978d-########-####-bc305be7c11e/VM/VM.vmx'] Consolidate Disks failed: N3Vim5Fault14TaskInProgress9ExceptionE(vim.fault.TaskInProgress)

    T02:22:35.985Z [50B87B90 warning 'vm:/vmfs/volumes/4eae978d-########-####-bc305be7c11e/VM/VM.vmx'] Failed operation

    T02:22:35.985Z [50B87B90 info 'TaskManager'] Task Completed : haTask-1-vim.VirtualMachine.consolidateDisks-459720263 Status error

    T02:22:35.985Z [50B87B90 info 'Default'] AdapterServer caught exception: vim.fault.TaskInProgress

    T02:22:36.085Z [51401B90 warning 'Hbrsvc'] Remove snapshot for vm (vmId=1) failed with fault vim.fault.TaskInProgress

  • Events tab of the virtual machine displays entries similar to:

    'VMname'in group 'Business Critical Protection Groups for SRM':Cannot resolve the file locations of the production VM for replication. error 6/9/2014 3:22:36PM


Environment

VMware vSphere ESXi 5.1

Cause

In vSphere 5.1, Microsoft Volume Shadow Copy (VSS) was introduced to replicate the quiesced state of the operating system or application. If quiescing is enabled for virtual machines running Windows 2003, 2008 or 2012, vSphere Replication creates and removes the virtual machine snapshot to create an application quiesced replica. If vSphere Replication and the backup software try to create or remove a snapshot at the same time, one process fails.

The virtual machine may have a backup snapshot. Whenever vSphere Replication or Host Based Replication (HBR) try to consolidate the disks, it does not add a new redo log to the disks. Later, the backup snapshot is removed and then, whenever HBR tries to consolidate the disks, it adds a new redo log to the disks.

This issue occurs because there is a parent disk open and HBR tries to consolidate the disks during replication.

Resolution

This is not a VMware issue. However, VMware has made code changes in ESXi 5.1 Update 2 to the improve the existing consolidation issue. 

There is a known interoperability issue between VMware vSphere Replication 5.1/5.5 and snapshot-based backup products. When a virtual machine is protected by vSphere Replication and it is also part of a snapshot-based backup job, snapshot conflicts can occur.

To avoid snapshot conflicts and related consolidation issues, perform one of these two options:
  • Ensure that vSphere Replication protected virtual machines which are also part of a backup schedule do not use quiescing, so that a snapshot will not be taken.

    Or

  • Use a backup solution that is not snapshot-based.


Additional Information

Powered by Wolken Software