SRM Test recovery fails at the IP customization phase with the message "Timed out waiting for VMware tools after 900 seconds"

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• Issue observed on Red Hat Enterprise Linux 8 vms

• These virtual machines are using guest managed open-vm tools

• The virtual machines on the production site has open-vm tools installed and running. This can be verified using the below commands
  rpm -qa | grep open-vm-tools
systemctl  status vmtoolsd

• The vmware tools timeout is already increased from the default timeout of 300 seconds to 900 seconds

• Reinstalling open vm tools using the below command and initiating test fails with the same error message
  dnf install -y open-vm-tools-12.3.5-2.e18

Cause

The issue occurs because the Guest OS fails to complete the boot process at the recovery site. The VMware Tools service requires a fully operational Operating System to initialize and establish communication. Consequently, the inability of the OS to boot prevents VMware Tools from starting, resulting in the observed timeout.

Cause Validation:

From the /opt/vmware/support/logs/srm/vmware-dr.log, below events are observed where in time out is observed waiting for the OS family to become known for VM tools when trying to power on the virtual machine as part of IP customization workflow

2025-11-07T11:44:27.520+05:30 error vmware-dr[02231] [SRM@6876 sub=Recovery ctxID=89c7f8a5 opID=c2b84921-xxxx-xxxx-xxxx-xxxxxxxxxxxx-test:xxxx:xxxx:xxxx:xxxx] [4cf975ed-xxxx-xxxx-xxxx-xxxxxxxxxxxx.ipCust-protected-vm-xxxxxx.waitForTools-vm-xxxxx] Timed out while Waiting for OS family to become known from VM tools; [vim.VirtualMachine:33765f57-xxxx-xxxx-xxxx-xxxxxxxxxxxx:vm-xxxxx]; timeout (secs): 900
2025-11-07T11:44:27.520+05:30 error vmware-dr[02231] [SRM@6876 sub=Recovery ctxID=89c7f8a5 opID=c2b84921-xxxx-xxxx-xxxx-xxxxxxxxxxxx-test:xxxx:xxxx:xxxx:xxxx] [4cf975ed-xxxx-xxxx-xxxx-xxxxxxxxxxxx.ipCust-protected-vm-xxxxxx] Unable to customize VM xxxxxxx [vm-xxxxx]; wait for tools failed: (dr.recovery.fault.ToolsTimedout) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>,
-->    timeout = 900
-->    msg = ""
--> }
--> [context]zKq7AVECAAQAAGFXdAEKdm13YXJlLWRyAAAsGRxsaWJ2bWFjb3JlLnNvAAFQYU9saWJkci1yZWNvdmVyeS5zbwABVR4dATRVTwIymQNsaWJmdW5jdGlvbmFsLnNvAADOKTQA0kI0AOB9SQOwjgBsaWJwdGhyZWFkLnNvLjAABN/6D2xpYmMuc28uNgA=[/context]

While manually powering on the virtual machine on the DR site on the isolated test network shows that the virtual machine fails to boot and enters emergency mode

Resolution

Engage guest OS team to validate why the OS is failing to boot.

Because the recovery VM is a replica, the issue may originate from the source itself. Schedule a maintenance window and reboot the VM at the production site. Verify if the Guest OS boots successfully and if VMware Tools initializes correctly. If the production VM also fails to boot, or if the issue persists solely at the recovery site, engage the Guest OS team.