IP Customization fails during a failover or test failover of a recovery plan in vCenter Site Recovery Manager

Products

VMware Live Recovery

Issue/Introduction

During a failover or test failover of a recovery plan in vCenter Site Recovery Manager (SRM), you experience these symptoms:

IP Customization fails for some or all virtual machines.
In the History tab under Recovery Plans, you see the error:

Error - Cannot complete customization, possibly due to a scripting runtime error or invalid script parameters (Error code: -1). IP settings may have been partially applied.
In the vmware-dr.log log file on the SRM server, you see entries similar to:

[00632 error 'Recovery' ctxID=27dcf3a7] [recovery-plan-3198.ipCust-protected-vm-3133-IPCustomization] Guest command on VM: [vim.VirtualMachine:vm-232] failed with exit code: 1 [02956 trivia 'Recovery' ctxID=27dcf3a7]<msg>Cannot complete customization, possibly due to a scripting runtime error or invalid script parameters (Error code: 1). IP settings may have been partially applied.</msg>
The virtual machine is powered off and the network interface is set to disconnected.
The recovery plan or test recovery plan completes with errors.

Note: This issue occurs with both Windows and Linux virtual machines. This issue can also show as a vix error codes = (1, 0) on Linux virtual machines.

Environment

VMware Site Recovery Manager 8.x
VMware Live Site Recovery 9.x

Cause

Windows

This issue occurs in Windows virtual machines if the default system variable for the %TEMP% or temporary folder within the guest operating system on the SRM server is changed from the default value and the new folder does not exist.

Note: The default folder specified in %TEMP% in Microsoft Windows is C:\windows\temp.

When you change the default value of %TEMP%, the guest operating system does not check if the folder exists. For example, if you change the default value of %TEMP% from C:\windows\temp to C:\temp, you do not see an error if C:\temp does not exist.

During Windows guest customization, SRM needs to write to the folder specified in %TEMP%. If this folder does not exist, guest customization fails.

Linux

This issue occurs in Linux virtual machines when the /tmp directory is mounted with the noexec option for security reasons.

During Linux guest customization, SRM needs to execute a file stored in /tmp. If this file cannot be executed, guest customization fails.

Note: Antivirus blocking access to temp folders will cause the issue

Resolution

Windows

To resolve the issue in a Windows virtual machine:

Ensure that the new directory for %TEMP% exists and can be written to.

Note: The %TEMP% variable is case sensitive. If it is set to C:\Windows\TEMP, but the real path is C:\Windows\Temp, guest customization will fail.

Linux

To resolve the issue in a Linux virtual machine:

Change the location of the IP customization script in the SRM advanced settings - SRM Recovery Advanced Settings
Set Value "recovery.defaultLinuxCustomizationBaseDir" to a folder on all linux VMs that has "executable" permissions - This change will need to be made under advanced settings for both SRM servers for a success failover and failback
You can verify if the noexec permission flag exists on a folder in the Linux guest by running command: findmnt -l | grep noexec

Additional Information

To fix issue on an individual Linux VM:

Allow files in the /tmp directory to be executed by removing the noexec option. This option can be removed by editing the /etc/fstab (as root) using 'vi' editor.

To work around this issue without changing the security options on the /tmp directory:

In the Linux virtual machine, open the VMware Tools initialization script using a text editor. The initialization script may be found in different locations depending on how VMware Tools was installed and configured on your system.
For systems using upstart (see 2015685 ), the initialization script will be one of the following:

* /etc/init/vmware-tools.conf. The command overview is in init(8) and the syntax is described in init(5).
* /etc/init/vmware-tools, which calls /etc/vmware-tools/services.sh

For systems that do not use upstart, the initialization script will be one of:

* /etc/init.d/vmware-tools
* /etc/init.d/vmware-tools-services.sh

One you have located the initialization script, add this line:

export TMPDIR=/new/desired/path
For example:
main()
{
export TMPDIR=/var/vmtmp

In this example, a new tmp directory called vmtmp was created. SRM will use this temp directory when applying guest customization.

Note: To specify an alternative temporary directory in RHEL7-based guests, add the following entry to the [Service] section of the systemd unit file for the service (usually in /usr/lib/systemd/system):

Environment="TMPDIR=/var/vmtmp"

Restart the VMware Tools service using whichever command is present on your system:

# /etc/init.d/vmware-tools restart
# /etc/init.d/vmware-tools-services.sh restart
# /etc/vmware-tools/services.sh restart

This forces a reload of the VMware Tools configuration file.

Note: Any modification to the VMware Tools initialization script will need to be reapplied when VMware Tools is upgraded.