SRM IP Customization fails with 'Failed to authenticate with the guest operating system using the supplied credentials'
Article ID: 397356
Updated On:
Products
Issue/Introduction
-
When we perform test recovery or planned migration it is observed that IP customization fails with the error "Failed to authenticate with the guest operating system using the supplied credentials"
-
Issue observed only for one linux vm (Red Hat Enterprise Linux 9)
-
There are other Red Hat RHEL vms in the same environment which works fine
-
Test takes 20 minutes per failed Virtual Machine to complete because SRM tries to customize the Guest OS 22 times before failing
Environment
VMware Live Site recovery 8.x
VMware Live Site recovery 9.x
Cause
Issue caused as the deployed Perl script is unable to locate necessary files, indicating issues in the Perl environment
From the recovery site /opt/vmware/support/logs/srm/vmware-dr.log file we can see that the IP customization is failing at CREATE_TEMP_FOLDER state with Invalid guest login.
2025-05-07T06:29:55.702Z warning vmware-dr[00999] [SRM@6876 sub=Recovery opID=cecf5fac-58b0-469e-bbc4-############-test:5012:763e:9231:7869:076e] [558360a4-505b-4c39-b440-############.ipCust-protected-vm-214290-IPCustomization] CREATE_TEMP_FOLDER_STEP Retrying (13 of 22) the guest operation after 60 seconds due to: (vim.fault.InvalidGuestLogin) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = "Received SOAP response fault from [<SSL(<io_obj p:0x00007f893c02a8d0, h:33, <TCP '10.#.#.#: 42104'>, <TCP '10.#.#.#: 443'>>), /sdk>]: createTemporaryDirectory
--> Failed to authenticate with the guest operating system using the supplied credentials."
--> }
--> [context]zKq7AVECAAQAANjOcAEKdm13YXJlLWRyAAAsGRxsaWJ2bWFjb3JlLnNvAIF3ixcBbGlidmltLXR5cGVzLnNvAAEoadwCKiEkbGlidm1vbWkuc28AAtMsJADOKTQA0kI0AOB9SQOwjgBsaWJwdGhyZWFkLnNvLjAABN/6D2xpYmMuc28u
To customize the IP, SRM needs to execute necessary customization scripts directly in the guest OS and the temp directory is the place where these scripts are injected and executed. SRM Solution user must authenticate against the guest OS before it can run any scripts. The Security Token Service issues a bearer token to SRM SU. Using STS signing certificates VMware tools checks if the issued bearer token is valid. In this case we can see that within the guest OS the token validation is failing.
/var/log/vmware-vgauhsvc.log file from the impacted vm gives the below events
[2025-05-07T06:39:56.994Z] [ warning] [VGAuthService] ServiceVerifyAndCheckTrustCertChainForSubject: no mapping entries or specified userName[2025-05-07T06:39:56.995Z] [ message] [VGAuthService] Returning error message ' <?xml version="1.0" encoding="UTF-8" ?><reply><sequenceNumber>1</sequenceNumber><errorCode>12</errorCode><errorMsg>validateSamlToken failed</errorMsg></reply>'
Further from the /var/log/vmware-imc/toolsDeployPkg.log we can see that the installed PERL is unable to locate necessary files, indicating issues in the Perl environment.================Perl script log end======================[2025-05-07T86:20:52.907Z] [ error] Customization command failed with exitcode: Z, stderr: 'Can't locate File/Find.pm in @INC (you may need to install the File::Find module) (@INC contains: /usr/local/lib64/per15/5.32 /usr/local/share/perl5/5.32 /usr/lib64/per15/vendor_perl /usr/share/per15/vendor_perl /usr/lib64/per15 /usr/share/per15) at /var/run/.vmware-imgcust-dPWC6Sq/guestals.pl line 12.BEGIN failed -- compilation aborted at /var/run/.vmware-imgcust-dPWC6Sq/guestals.pl line 12.
Resolution
To resolve this issue download and install the latest PERL software on this vm
-
Verify if PERL is installed on the virtual machine using the command
perl -v. Linux guest OS requires PERL to be installed as scripts are launched using PERL
Additional Information
You can also use the following command to verify the perl modules that are loaded to compare against the error from toolsDeployPkg.log
perl -e 'while (<@INC>) { while (<$/*.pm>) { print "$\n"; } }'Feedback
