SRM IP Customization - Failed to authenticate with the guest operating system using the supplied credentials

Products

• VMware Live Recovery

Issue/Introduction

Symptoms:
Error - Received SOAP response fault from [<cs p:00007fc938030bb0, TCP: vcsa67pr.vr.local:443>]: createTemporaryDirectory

Failed to authenticate with the guest operating system using the supplied credentials.

Test takes a long time to complete because SRM tries to customize the Guest OS 22 times before failing

OS             : Cent OS 7
VM Tools : 11.0.5.17716, Build-15389592

vmware-dr.log:

2021-03-15T13:07:05.805Z info vmware-dr[31630] [SRM@6876 sub=Recovery opID=d0b14b2d-e691-44dd-8154-2a6e844863e1-test:270f:7d82:31bb:ce5b:2e73:2d87] [4dc409ae-9c9f-4fd7-9bff-c5d0d0324399
.ipCust-protected-vm-175843-IPCustomization]  CREATE_TEMP_FOLDER_STEP  Trigger guest operation execution. Attempt (10 of 22)
2021-03-15T13:07:05.865Z warning vmware-dr[14696] [SRM@6876 sub=Recovery opID=d0b14b2d-e691-44dd-8154-2a6e844863e1-test:270f:7d82:31bb:ce5b:2e73:2d87] [4dc409ae-9c9f-4fd7-9bff-c5d0d0324
399.ipCust-protected-vm-175843-IPCustomization]  CREATE_TEMP_FOLDER_STEP  Retrying (11 of 22) the guest operation after 60 seconds due to: (vim.fault.InvalidGuestLogin) {
-->    faultCause = (vmodl.MethodFault) null,
-->    faultMessage = <unset>
-->    msg = "Received SOAP response fault from [<cs p:00007fc938030bb0, TCP:host.example.com:443>]: createTemporaryDirectory
--> Failed to authenticate with the guest operating system using the supplied credentials."
--> }
--> [context]zKq7AVECAAQAANKIAQELdm13YXJlLWRyAACu1xZsaWJ2bWFjb3JlLnNvAAHjRuVsaWJ2aW0tdHlwZXMuc28AgYHjBAECCn0QbGlidm1vbWkuc28AAux6GAJUhRgAtZ4rALjWKwAP3TgDlXQAbGlicHRocmVhZC5zby4wAAQPAw9s
aWJjLnNvLjYA[/context]
--> [backtrace begin] product: VMware vCenter Site Recovery Manager, version: 8.3.1, build: build-16877778, tag: vmware-dr, cpu: x86_64, os: linux, buildType: release
--> backtrace[03] libvmacore.so[0x0016D7AE]
--> backtrace[04] libvim-types.so[0x00E546E3]
--> backtrace[05] libvim-types.so[0x0104E381]
--> backtrace[06] libvmomi.so[0x00107D0A]
--> backtrace[07] libvmomi.so[0x00187AEC]
--> backtrace[08] libvmomi.so[0x00188554]
--> backtrace[09] libvmacore.so[0x002B9EB5]
--> backtrace[10] libvmacore.so[0x002BD6B8]
--> backtrace[11] libvmacore.so[0x0038DD0F]
--> backtrace[12] libpthread.so.0[0x00007495]
--> backtrace[13] libc.so.6[0x000F030F]
--> [backtrace end]

Guest OS logs:

/var/log/vmware-vgauthsvc.log:

[2021-03-15T18:32:16.446Z] [   debug] [VGAuthService] SendRpciPacket: RPC returned 'Unknown command'
[2021-03-15T18:32:16.446Z] [   debug] [VGAuthService] Pref_GetString: Pref_GetString(aliasStoreDir) failed: Key file does not have key “aliasStoreDir” in group “service”
[2021-03-15T18:32:16.446Z] [ message] [VGAuthService] Using '/var/lib/vmware/VGAuth/aliasStore' for alias store root directory
[2021-03-15T18:32:16.451Z] [   debug] [VGAuthService] Pref_GetInt: Pref_GetInt(ticketTTL) failed: Key file does not have group “ticket”
[2021-03-15T18:32:16.451Z] [   debug] [VGAuthService] ServiceInitTicketPrefs: ticket TTL set to 86400 seconds
[2021-03-15T18:32:16.451Z] [   debug] [VGAuthService] ServiceInitTicketPrefs: computed reapCheckTime as 8640 seconds
[2021-03-15T18:32:16.460Z] [ message] [VGAuthService] LoadCatalogAndSchema: Using '/etc/vmware-tools/vgauth/schemas' for SAML schemas
[2021-03-15T18:32:16.489Z] [   debug] [VGAuthService] Pref_GetInt: Pref_GetInt(clockSkewAdjustment) failed: Key file does not have key “clockSkewAdjustment” in group “service”
[2021-03-15T18:32:16.489Z] [ message] [VGAuthService] LoadPrefs: Allowing 300 of clock skew for SAML date validation
[2021-03-15T18:32:16.489Z] [ message] [VGAuthService] SAML_Init: Using xmlsec1 1.2.20 for XML signature support
[2021-03-15T18:32:16.490Z] [   debug] [VGAuthService] SendRpciPacket: RPC returned 'Unknown command'
[2021-03-15T18:32:16.490Z] [   debug] [VGAuthService] Pref_GetInt: Pref_GetInt(listenTTL) failed: Key file does not have key “listenTTL” in group “service”
[2021-03-15T18:32:16.490Z] [   debug] [VGAuthService] ServiceInitListenConnectionPrefs: listen conn TTL set to 1800 seconds
[2021-03-15T18:32:16.490Z] [   debug] [VGAuthService] ServiceInitListenConnectionPrefs: computed reapCheckTime as 180 seconds
[2021-03-15T18:32:16.490Z] [   debug] [VGAuthService] Pref_GetInt: Pref_GetInt(maxDataConnectionsPerUser) failed: Key file does not have key “maxDataConnectionsPerUser” in group “service”

/var/log/vmware-imc/toolsDeployPkg.log:

[2021-03-15T16:20:36.344Z] [   error] Customization command failed with exitcode: 2, stderr: 'Can't locate Exporter.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /usr/share/perl5/File/Basename.pm line 54.
Compilation failed in require at /var/run/.vmware-imgcust-dvbLmL6/guestals.pl line 11.
BEGIN failed--compilation aborted at /var/run/.vmware-imgcust-dvbLmL6/guestals.pl line 11.
'.

[2021-03-15T16:20:36.344Z] [   error] Customization process returned with error.
[2021-03-15T16:20:36.344Z] [   debug] Deployment result = 2.
[2021-03-15T16:20:36.344Z] [    info] Setting 'unknown' error status in vmx.
[2021-03-15T16:20:36.345Z] [    info] Transitioning from state 'INPROGRESS' to state 'ERRORED'.
[2021-03-15T16:20:36.345Z] [    info] ENTER STATE 'ERRORED'.
[2021-03-15T16:20:36.345Z] [    info] EXIT STATE 'INPROGRESS'.
[2021-03-15T16:20:36.345Z] [   debug] Setting deploy error: 'Deployment failed.The forked off process returned error code.'.
[2021-03-15T16:20:36.345Z] [   error] Deployment failed.The forked off process returned error code.
[2021-03-15T16:20:36.345Z] [    info] Launching cleanup.
[2021-03-15T16:20:36.345Z] [   debug] Command to exec : '/bin/rm'.
[2021-03-15T16:20:36.345Z] [    info] sizeof ProcessInternal is 56
[2021-03-15T16:20:36.345Z] [    info] Returning, pending output from stdout
[2021-03-15T16:20:36.345Z] [    info] Returning, pending output from stderr
[2021-03-15T16:20:36.347Z] [    info] Process exited normally after 0 seconds, returned 0
[2021-03-15T16:20:36.347Z] [    info] No more output from stdout
[2021-03-15T16:20:36.347Z] [    info] No more output from stderr
[2021-03-15T16:20:36.347Z] [    info] Customization command output: ''.
[2021-03-15T16:20:36.347Z] [    info] sSkipReboot: 'false', forceSkipReboot 'true'.
[2021-03-15T16:20:36.347Z] [   error] Deploy error: 'Deployment failed.The forked off process returned error code.'.
[2021-03-15T16:20:36.347Z] [   error] Package deploy failed in DeployPkg_DeployPackageFromFile
[2021-03-15T16:20:36.347Z] [   debug] ## Closing log

Environment

VMware Site Recovery Manager 8.3.x

Resolution

Guest OS Customization Support Matrix - http://partnerweb.vmware.com/programs/guestOS/guest-os-customization-matrix.pdf

Cent OS 7 is supported but the current Perl version is perl5

Download and install the latest Perl software


For Windows follow the steps in this kb.

https://kb.vmware.com/s/article/93823