Test failover/failover operation in VMware vCenter Site Recovery Manager fails at the Change Network Settings step with the error: Timed Out

Products

• VMware

Issue/Introduction

Symptoms:

When performing a test failover or fail over operation in VMware vCenter Site Recovery Manager (SRM), you experience these symptoms:

• Test failover fails to complete
• Failover fails to complete
• Guest customization fails during the Changing Network Settings step
• You see one or more of these errors:
  • Timed Out
  • Error - Timed out waiting for Customization scripts to complete in 600 seconds. IP settings may have been partially applied.
  • Error: Timed out waiting for VMware Tools after 300 seconds

Cause

This issue may occur if the network settings for the virtual machine’s guest operating system is customized as a part of recovery. This error indicates that the virtual machine failed to report the new network settings within the default timeout period of 300 seconds.

Resolution

To troubleshoot and resolve this issue:

1. Ensure that the latest version of VMware Tools is installed on all protected virtual machines. VMware Tools is required for IP customization. For more information, see Verifying a VMware Tools build version (1003947).
2. IP customization does not work on bubble network (portgroup with no network cards). Ensure that you choose a recovery network.
3. Edit the recovery plan and increase the Change Network Settings timeout value. The default value is 300 seconds and the maximum is 900 seconds.
   
   For specific virtual machines showing the error:
   
   
   1. Right-click virtual machine in the recovery plan and select Configure.
   2. Go to Startup Action > Power On > Wait for VMware tools.
   3. Change from the default value of 5 minutes to 15 minutes.
      
4. If you see this message:
   
   Error - Timed out waiting for Customization scripts to complete in 600 seconds. IP settings may have been partially applied.
   
   Modify the value of the recovery.customizationTimeout parameter from the default value of 600 to a higher value.
   
   To reset the value of the recovery.customizationTimeout parameter:
   
   
   1. Right-click the site and click Advanced Settings > Recovery.
   2. Modify the value of the recovery.customizationTimeout parameter from the default value of 600 to a higher value.
      
      Notes:
      • The minimum and maximum allowed values are 0 and 2147483647 respectively.
      • Repeat this step for other site also.
      • You must restart SRM services on both sites for the changes to take effect.
        
        
5. If the virtual machine is configured as a DHCP client, verify that a DHCP server is available on the recovery network.
6. If the virtual machine is configured with a static IP address, ensure that the address is available and is not in use on the recovery network.
7. For more information about the errors that prevented the failover or test failover operation from completing, review the log files created by the customization process on each virtual machine.
   
   The log files are located at:
   
   
   • Windows virtual machines – C:\windows\temp\vmware-imc\toolsDeployPkg.txtIn
   • Linux virtual machines – /var/log/vmware-imc/toolsDeployPkg.txt
   
   
8. vCenter Server at the recovery site may be attempting to power on too many virtual machines at the same time. For more information, see Powering on many virtual machines simultaneously on the recovery site can lead to errors (2010593).

Note: If the problem still exists after trying the steps in this article:

• Gather the VMware Support Script Data. For more information, see Collecting diagnostic information for Site Recovery Manager (1009253).
• File a support request with VMware Technical Support and note this Article ID in the problem description. For more information, see Filing a Support Request in Customer Connect (2006985).

Additional Information

Verifying a VMware Tools build version
Collecting diagnostic information for Site Recovery Manager
How to file a Support Request in Customer Connect
Powering on many virtual machines simultaneously on the recovery site can lead to errors
VMware vCenter Site Recovery Manager でのフェイルオーバーのテスト操作/フェイルオーバー操作がネットワーク設定の変更手順で次のエラーで失敗する：タイムアウトになりました
在 VMware vCenter Site Recovery Manager 中执行测试故障切换/故障切换操作时在“更改网络设置”步骤失败，并显示错误消息：超时