Test recovery for Linux VMs fails during the IP customization stage with the error "Failed to authenticate with the guest operating system using the supplied credentials"

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:

• During a Test failover, the recovery plan fails at the stage "Customize IP".
• The SRM interface displays the following error message:
  "Failed to authenticate with the guest operating system using the supplied credentials"
• This symptom is seen on Linux-based Virtual Machines.

Environment

VMware Live Site Recovery 9.x

 

Cause

• The "/var/log" directory within the Guest OS is 100% full.
• When SRM initiates IP customization, the VMware tools attempts to create a temporary directory "/var/log/vmware-imc". If the filesystem is at maximum capacity, the mkdir command fails. Because the temporary directory cannot be created, the subsequent guest authentication and script deployment tasks cannot proceed, resulting in an "Failed to authenticate" error in the SRM interface.

Cause Validation:

• The /opt/vmware/support/logs/srm/vmware-dr.log file on the target SRM confirms that the IP customization failed while creating temp directory.
  YYYY-MM-DDTHH:MM.SSSZ error vmware-dr[04340] [SRM@6876 sub=Recovery ctxID=######## opID=########-####-####-####-############-test:####:####:####] [########-####-####-####-############.failoverOrchJob] IP customization failed for VM ####### [vm-####]: (vim.fault.InvalidGuestLogin) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<SSL(<io_obj p:0x#########, h:##, <TCP '##.###.##.## : 38054'>, <TCP '##.###.##.##: 443'>>), /sdk>]: createTemporaryDirectory
--> Failed to authenticate with the guest operating system using the supplied credentials."
--> }
  
  
• Login to Guest OS. Navigate to "/var/log". The "vmware-imc" folder do not appear.
  cd /var/log/
ls -1 | grep -i vmware-imc
  
  
• The "/var/log/vmware-vmsvc-root.log" file inside the Guest confirms that "vmware-imc" directory is not created since there is no free space. 
  [YYYY-MM-DDTHH:MM.SSSZ] [ warning] [deployPkg] [1144] Customization command failed with exitcode: 127, stderr: ''.
[YYYY-MM-DDTHH:MM.SSSZ] [ warning] [deployPkg] [1144] Customization command failed with stderr: '/bin/mkdir: cannot create directory ‘/var/log/vmware-imc’: No space left on device
  
  
• Running the command "df -h" inside the guest confirms that the /var/log directory is100% full.
  df -h
Filesystem                 Size   Used   Avail   Use%   Mounted on
/dev/mapper/rhel-var_log   10G    10G    20K     100%   /var/log

Resolution

Engage Guest OS team and free up the space at "/var/log" directory.