VM Recovery Error while DR Drill causing incomplete recovery of VM and failed with error 'Timed out waiting for VMware Tools after 300 seconds'

Products

• VMware vSphere ESXi

Issue/Introduction

Symptom:

• The failover did not complete and failed with the error :
  
  
  • The recovery operation encountered errors. Review the plan history to view any errors or warnings. Once the cause of the errors has been fixed, you can press Recovery again. Virtual machines that have already been successfully recovered will not be affected by running the recovery again.
    
    
  • Timed out waiting for VMware Tools after 300 seconds
    
    

Environment

VMware Live Site Recovery 9.0

Cause

• VMware Live Site Recovery uses VMware Tools heartbeat to discover when recovered virtual machines are running on the recovery site. Recovery operations require that you install VMware Tools on the protected virtual machines. Recovery fails if you did not install VMware Tools on the protected virtual machines.

 From /var/log/vmware/srm/vmware-dr.log shows repeated timeout errors,

####-##-##T14:02:02.404+05:30 error vmware-dr [01336] [SRM@6876 sub=Recovery ctxID=f02dfadd opID=2e####2f-####-####-####-b3########b2-failover:a0eb:####:####:225b-powerOffVm: 6d##: a9##] Wait for shutdown failed for VM ' [vim.VirtualMachine:2c####a9-####-####-####-21########ae:vm-10]'', bailing out since stop on errors is set. Error: N2Dr16TimeoutExceptionE Operation timed out after '300.000000' seconds
####-##-##T14:02:02.404+05:30 info vmware-dr[01336] [SRM@6876 sub=DrTask ctxID=f02dfadd opID=2e####2f-####-####-####-b3########b2-failover:a0eb:####:####:225b-powerOffVm: 6d##: a9##] Task 'dr.recovery. RecoveryManager.powerOffVm4' failed

 

Resolution

Recommended Fix : Install the VMware tools on the impacted VMs and then perform the recovery. 

Note - If the issue persists despite VMware Tools being active, increase the Startup Action wait time to 900 seconds (15 minutes). The default timeout of 300 seconds (5 minutes) may be insufficient for the Open VM Tools/VMware Tools service to fully initialize and establish a heartbeat with the host, causing the recovery step to fail.

 

Workaround: 

If installing the VMware tools option is not possible for the VM, then update the VMware Live Site Recovery plan not to wait for VMware Tools to start in the recovered virtual machines.

• On the Live Site Recovery home tab, select a site pair, and click View Details.
• Select the Recovery Plans tab, and click on the recovery plan to which the virtual machine belongs.
• On the Virtual Machines tab, right-click a virtual machine and click Configure Recovery.
• Make the changes you want to the recovery properties settings.
• Click OK.

Refer - Recovery Settings to Virtual Machines in a Recovery Plan

Before updating the VM recovery properties :

After updating the VM recovery properties :

Additional Information

Recovery Plan Times Out While Waiting for VMware Tools 

Change Recovery Settings