Error: "Timed out waiting for VMware tools after 900 seconds" during SRM Test Recovery

Products

VMware Live Recovery

Issue/Introduction

During a VMware Site Recovery Manager (SRM) Test Recovery, virtual machines successfully execute the "Power On" step but fail shortly after. The recovery plan terminates with a timeout error, preventing the VM to run at the recovery site.

Symptoms:

SRM Task: Power On VMs (or similar) fails.
Error Message: Timed out waiting for VMware tools after 900 seconds.
vmware-dr.log shows: Wait for VM Tools in VM <VM_NAME> failed: (dr.recovery.fault.ToolsTimedout).
The VMware tools timeout is already increased from the default timeout of 300 seconds to 900 seconds.

Environment

VMware Site Recovery Manager (SRM) 9.x

VMware Tools (Any Version)

Cause

The failure is caused by the Guest Operating System (OS) hanging or experiencing a delayed boot sequence at the recovery site. Because the OS fails to initialize its kernel or services, the VMware Tools daemon (vmtoolsd) never starts. Without vmtoolsd, the GuestRPC channel which is used to communicate the "OS Family" and "Heartbeat" to the SRM is never established.

1. SRM Timeline (vmware-dr.log)

The SRM logs show the 15-minute (900s) watchdog timer starting and ending precisely on schedule. Note that SRM is waiting specifically for the "OS Family" to be reported, which requires the Guest OS to be fully initialized.

- Hardware power on completes and SRM starts the wait period.
  
  2026-02-10T10:50:48.826Z info vmware-dr[320155] [SRM@6876 sub=Recovery ctxID=##.masterPowerOn-protected-vm-######] Power on succeeded for VM VM_NAME[vm-######]
  .
  .
  2026-02-10T10:50:48.831Z info vmware-dr[355922] [SRM@6876 sub=Recovery ctxID=##.waitForTools-vm-#####] Waiting for OS family to become known from VM [vim.VirtualMachine:##:vm-######]; timeout: 900 seconds
- SRM hits the 900-second limit and terminates the task.
  
  2026-02-10T11:05:48.832Z error vmware-dr[320150] [SRM@6876 sub=Recovery ctxID=##.waitForTools-vm-#####] Timed out while Waiting for OS family to become known from VM tools; [vim.VirtualMachine:##:vm-######]; timeout (secs): 900
  
  2026-02-10T11:05:48.832Z error vmware-dr[320168] [SRM@6876 sub=Recovery ctxID=##.masterPowerOn-protected-vm-######] Wait for VM Tools in VM VM_NAME[vm-######] failed: (dr.recovery.fault.ToolsTimedout) {
  --> faultCause = (vmodl.MethodFault) null,
  --> faultMessage = <unset>,
  --> timeout = 900
  --> msg = ""

2. VM Kernel & Heartbeat Timeline (vmware.log)

The VM logs reveal that while the hardware powered on, the Guest OS failed to establish communication with the host.

- Tools Status and state change.
  
  2026-02-10T10:50:48.721Z In(05) vmx - Tools: sending 'OS_PowerOn' (state = 3) state change request
  2026-02-10T10:50:48.721Z In(05) vmx - Tools: Delaying state change request to state 3.
  2026-02-10T10:50:48.721Z In(05) vmx - TOOLS INSTALL initializing state to IDLE on power on.
  - The logs (Tools: sending 'OS_PowerOn' (state = 3) state change request) means that the Virtual Machine Monitor (VMM) is trying to send a signal into the Guest OS saying, "The hardware is powered on, please start the VMware Tools services."
  - Immediately we see the next message "Tools: Delaying state change request to state 3.". The host must wait for the Windows/Linux kernel to load the vmtoolsd drivers before it can actually complete this state change. If the OS hangs during boot, we will see this "Delaying" message, and the state change will never actually finish.
  - The last message "TOOLS INSTALL initializing state to IDLE on power on." means that the host is resetting the "Auto-Update" or "Install Tools" wizard status. It sets it to IDLE to ensure that it doesn't try to force an installation or upgrade while the VM is still trying to perform its initial boot at the recovery site.
- Critical Failure Point. The ESXi host gives up after 10 minutes (600 seconds) of zero heartbeat/activity from the Guest OS.
  
  2026-02-10T11:00:48.749Z In(05) vmx - Tools: No activity for 10 minutes, resetting Tools version.
  2026-02-10T11:00:48.749Z In(05) vmx - TOOLS setting legacy tools version to '0' type 0, manifest status is 4
  2026-02-10T11:00:48.749Z In(05) vmx - DDB: "toolsVersion" = "0" (was "13317")
  - The message "Tools: No activity for 10 minutes, resetting Tools version." reflect that the ESXi hypervisor has an internal watchdog timer for VMware Tools. If a VM is powered on but the host receives zero heartbeat packets and zero RPC (Guest Operations) traffic for exactly 600 seconds (10 minutes), the host concludes that the guest OS has failed to load the tools service

Resolution

The resolution requires shifting focus from the SRM application to the Guest OS boot health.

Direct Console Observation:
- Perform a Test Recovery and immediately open the VM Console.
- Monitor for OS-level hangs, such as:
  - Windows: Stuck on the "Getting devices ready" or update screen.
  - Linux: Stuck at a fsck prompt or kernel panic (e.g., "Wait for root device").
- Address the specific OS-level failure (e.g., repairing the boot loader or fixing driver conflicts).
Modify SRM VMware Tools Timeout:
- If the OS is healthy but simply slow to boot (e.g., heavy database initialization), increase the SRM timeout.
- In the SRM Interface, go to Recovery Plan > Recovery Steps.
- Right-click the affected VM > Configure Recovery > Recovery Priority.
- Change the VMware Tools timeout from 900 to 1800 seconds.

Additional Information

SRM Test recovery fails at the IP customization phase with the message "Timed out waiting for VMware tools after 900 seconds"

VM Recovery Error while DR Drill causing incomplete recovery of VM and failed with error 'Timed out waiting for VMware Tools after 300 seconds'

Recovery Plan Times Out While Waiting for VMware Tools

Change Recovery Settings