IP Customization Failure During RHEL VM Recovery in SRM with Error Code 126 or Error Code 255

Products

VMware Live Recovery VMware Cloud on AWS

Issue/Introduction

Symptoms

During the recovery of a RHEL (Red Hat Enterprise Linux) virtual machine using Site Recovery Manager, you may encounter the following error message:

“Cannot complete customization, possibly due to a scripting runtime error or invalid script parameters (Error code: 126 or 255). IP settings may have been partially applied.”

This error typically occurs during the IP customization phase of the recovery process, where SRM attempts to configure the IP settings of the VM.
"vmware-dr.log", may show an error message similar to the following:

--> 2023-10-15T09:04:08 DEBUG: opening file for writing (/etc/resolv.conf). --> 2023-10-15T09:04:08 DEBUG: Permission denied writing data to file /etc/resolv.conf.--> 2023-10-15T09:04:08 INFO: File Attributes: /etc/resolv.conf --> 2023-10-15T09:04:08 DEBUG: Command: 'lsattr /etc/resolv.conf'

The critical part of this log indicates a "Permission denied" error when attempting to write to the /etc/resolv.conf file. This is followed by an error code 255 or 126, signaling the failure of IP customization.

--> Exit code: 255
--> ",
--> returnValue = 255
--> }
--> msg = "Cannot complete customization, possibly due to a scripting runtime error or invalid script parameters (Error code: 255). IP settings may have been partially applied."
--> },
--> (dr.recovery.fault.IPCustomizationFailure) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> result = (dr.recovery.CalloutResult) {
--> commandLine = "echo "Starting IP customization on Linux ..." >> "$VMware_GuestOp_OutputFile"

OR

--> " <_type>dr.recovery.fault.IPCustomizationFailure</_type>",
--> " <msg>Cannot complete customization, possibly due to a scripting runtime error or invalid script parameters (Error code: 126). IP settings may have been partially applied.</msg>",
--> " <result>",
--> " <_type>dr.recovery.CalloutResult</_type>",
--> " <commandLine>"echo "Starting IP customization on Linux ..." >> "$VMware_GuestOp_OutputFile"",

Further examination of the /var/log/vmware-imc/toolsDeployPkg.log in the guest OS reveals the following error messages related to the issue:

2825-01-08T14:26:51 ERROR: Fatal error occurred during customization !! Customization halted.
2025-01-08T14:26:51 ERROR: Error: Operation not permitted:Error writing data to File (/etc/resolv.conf). Operation not permitted at /var/run/.vmware-imgcust-d0ox8Vv/scripts/Utils.pm line 656.
2825-01-08T14:28:51 INFO: Return code is 252.
2025-01-00714:26:51 DEBUG: Removing lock file /var/lock/ure/gosc

This error indicates that the IP customization failed due to insufficient permissions to modify the /etc/resolv.conf file, which is crucial for network configuration during recovery.

Environment

VMware Site Recovery Manager 8.x
VMware Site Recovery Manager 9.x
Guest OS - RHEL (Red Hat Enterprise Linux)

Cause

The root cause of the issue is that the /etc/resolv.conf file in the RHEL 8 guest OS has the immutable flag set, which prevents any modifications to the file. This flag is preventing SRM from making the necessary changes to the file during the IP customization process.

You can verify this by running the following command in the guest OS:

# lsattr /etc/resolv.conf

If the immutable flag is set, the output will display an "i" in the attribute column:

----i----------- /etc/resolv.conf

The "immutable" flag in Linux means that the file cannot be modified, even by the root user. This is typically set using the chattr command and is used to protect critical system files from modification.

Resolution

To resolve the issue, follow these steps:

Remove the Immutable Flag:

1. To allow modifications to the file, remove the immutable flag using the following command:

# sudo chattr -i /etc/resolv.conf

2. After running this command, verify that the flag has been removed by checking the attributes again:

# lsattr /etc/resolv.conf

3. The output should now look like this, with no "i" character:

# lsattr /etc/resolv.conf
---------------------- /etc/resolv.conf
Sync the changes made on the virtual machine. If you are using vSphere replication, click on sync now to sync the recent changes. If you are using Array Based replication, ensure that the changes made are synced.
Retry the Recovery Process
If you still observe the same permission errors after making the above changes, engage your guest OS vendor to investigate further. In some cases a security script will be running within the guest OS which will overwrite the permissions you have set.

Additional Information

If you are doing this on a VM that is part of a Site Recovery Manager (SRM) plan,do not set this flag on the Source/Protected VM if you plan to run Recovery or Test plans frequently. It will cause the same Operation not permitted error you just fixed.

To set the immutable flag back on (locking the file again), you use the same chattr command but with a + instead of a -.

Here is the procedure:

1. Set the Immutable Flag

Run this command as root:

# sudo chattr +i /etc/resolv.conf

+i: Adds the immutable attribute.

2. Verify the Lock

Run this command to confirm the flag is active:

# lsattr /etc/resolv.conf

Expected Output: ----i--------e-- /etc/resolv.conf (The presence of the i confirms the file cannot be modified, deleted, or renamed.)

Best Practice to follow, if your security policy requires this flag, the best place to run this command is in a Post-Customization Script.

You can create a script in /etc/vmware-tools/scripts/ or add a "Post-Power On" step in the SRM Recovery Plan to run chattr +i /etc/resolv.conf after the IP customization has successfully finished.