Powering on many virtual machines simultaneously on the recovery site can lead to errors

Products

• VMware Live Recovery

Issue/Introduction

• When powering on many virtual machines simultaneously on the recovery site, you might see these errors in the recovery history reports:
  
• Error - The command 'echo "Starting IP customization on Windows ..." > > % VMware_GuestOp_OutputFile%.
• Error - Cannot complete customization, possibly due to a scripting runtime error or invalid script parameters.
• Error - An error occured when uploading files to the guest VM.
• Error - Timed out waiting for VMware Tools after 300 seconds.
  
  
• These errors might occur during both array-based and vSphere Replication recovery. The errors are caused because too many virtual machines perform boot operations at the same time.

By default, SRM does not limit the number of power-on operations that can be performed simultaneously. If you encounter errors while virtual machines power on on the recovery site, you can modify the file vmware-dr.xml file to set a limit on the number of virtual machines that power on simultaneously.

Environment

VMware vCenter Site Recovery Manager 5.0.x

Resolution

If you encounter such errors, you should limit the number of power-on operations on the recovery site according to the capacity of your environment. You can limit the number of power on operations either for a standalone host or for a cluster.

This example limits the number of power-on operations to a maximum of 32 per cluster and 4 per standalone host:

1. Go to C:\Program Files\VMware\VMware vCenter Site Recovery Manager\config on the recovery server.
2. Open the file vmware-dr.xml in a text editor.
3. Add these lines in the <config> section:
   
   <defaultMaxBootAndShutdownOpsPerCluster>32</defaultMaxBootAndShutdownOpsPerCluster>
<defaultMaxBootAndShutdownOpsPerHost>4</defaultMaxBootAndShutdownOpsPerHost>
   
   
4. Restart the SRM Server service.

For related information, see The test failover/failover operation in VMware vCenter Site Recovery Manager fails at the Change Network Settings step with the error: Timed Out(1031107).

Alternatively, remove and re-protect the affected virtual machines to resolve this issue.