Scheduled File-Based Backup fails to start on vCenter server
search cancel

Scheduled File-Based Backup fails to start on vCenter server

book

Article ID: 402458

calendar_today

Updated On:

Products

VMware vCenter Server VMware vCenter Server 8.0

Issue/Introduction

  • Schedule file-based backup job fails to start or trigger on vCenter server.

  • Manual file-based backup works fine on vCenter server.

  • When trying to configure or edit the VAMI backup returns error: "Path not exported by the remote filesystem."

  • From /var/log/vmware/applmgmt/backupScheduler.log, following errors are observed

    YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-#######] [VapiClientHelper::get_saml_token_with_svc_user:VapiClientHelper.py:117] ERROR: Failed to get HOK token with error SoapException:
faultcode: ns0:FailedAuthentication
faultstring: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: <Example.com>}
faultxml: <?xml version='1.0' encoding='###-#'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: <Example.com>}</faultstring></S:Fault></S:Body></S:Envelope>.
YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-#######] [VapiClientHelper::get_vapi_stub_with_saml_auth:VapiClientHelper.py:92] ERROR: Failed to getting vapi stub with svc user: name 'svcUtil' is not defined
YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-#######] [Scheduler::ExecScheduleRun:Scheduler.py:137] ERROR: Failed to issue the Schedules.run request. Exception: {messages : [LocalizableMessage(id='com.vmware.applmgmt.backup.plugin.fs_path_not_found', default_message='Path not exported by the remote filesystem.', args=['Plugin error occurred. ErrCode: 151, Args: ()'], params=None, localized=None)], data : None, error_type : None}
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/Scheduler.py", line 133, in ExecScheduleRun
    status = svc_handle.run(scheduleId, comment='SCHEDULED')
  File "/usr/lib/applmgmt/pyclient/applmgmt_client-1.0-py2.7.egg/com/vmware/appliance/recovery/backup_client.py", line 1197, in run
    return self._invoke('run',
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 295, in native_invoke
    raise TypeConverter.convert_to_python(method_result.error,  # pylint: disable=E0702
com.vmware.vapi.std.errors_client.Error: {messages : [LocalizableMessage(id='com.vmware.applmgmt.backup.plugin.fs_path_not_found', default_message='Path not exported by the remote filesystem.', args=['Plugin error occurred. ErrCode: 151, Args: ()'], params=None, localized=None)], data : None, error_type : None}

  • From /var/log/vmware/sso/vmware-identity-sts.log following errors are observed

    YYYY-MM-DDTHH:MM:SS WARN sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.interop.ldap.LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.OpenLdapClientLibrary, error code: 49
YYYY-MM-DDTHH:MM:SS WARN sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.ServerUtils] cannot bind connection: [ldap://VCENTER.FQDN:389, vmware-applmgmtservice-########-####-####-####-############@vsphere.local]
YYYY-MM-DDTHH:MM:SS ERROR sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.ServerUtils] cannot establish ldap connection with URI: [ldap://<FQDN_OF_vCENTER>:389] because [Invalid credentials] therefore will not attempt to use any secondary URIs
YYYY-MM-DDTHH:MM:SS WARN sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider] Failed to authenticate using SRP binding
com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials
YYYY-MM-DDTHH:MM:SS WARN sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider] The user is ###-####### and failed to authenticate.
YYYY-MM-DDTHH:MM:SS ERROR sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.IdentityManager] Failed to authenticate principal [vmware-applmgmtservice-########-####-####-####-############@vsphere.local] for tenant [vsphere.local]
javax.security.auth.login.LoginException: Login failed
 at com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider.authenticate(VMwareDirectoryProvider.java:428) ~[libvmware-identity-idm-server.jar:?]

YYYY-MM-DDTHH:MM:SS INFO sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.diagnostics.VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[vsphere.local], eventid=[USER_NAME_PWD_AUTH_FAILED], level=[ERROR], category=[VMEVENT_CATEGORY_STS], text=[ParameterizedMessage[messagePattern=Failed to authenticate principal [{}]. User password expired., stringArgs=[vmware-applmgmtservice-########-####-####-####-############@vsphere.local], throwable=null]], detailText=[null], corelationId=[########-####-####-####-############], timestamp=[##########]
YYYY-MM-DDTHH:MM:SS ERROR sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.IdentityManager] Failed to authenticate principal [vmware-applmgmtservice-########-####-####-####-############@vsphere.local]. User password expired.
YYYY-MM-DDTHH:MM:SS INFO sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.IdentityManager] Authentication failed for user [vmware-applmgmtservice-########-####-####-####-############@vsphere.local] in tenant [vsphere.local] in [20] milliseconds with provider [vsphere.local] of type [com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider]
YYYY-MM-DDTHH:MM:SS ERROR sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.ServerUtils] Exception 'com.vmware.identity.idm.PasswordExpiredException: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}'
com.vmware.identity.idm.PasswordExpiredException: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}
 at com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider.checkUserAccountFlags(VMwareDirectoryProvider.java:1458) ~[libvmware-identity-idm-server.jar:?]
 at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:3159) ~[libvmware-identity-idm-server.jar:?]
YYYY-MM-DDTHH:MM:SS INFO sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:FailedAuthentication and description: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}
  • From /var/log/vmware/vmdird/vmdird.log following errors are observed
    YYYY-MM-DDTHH:MM:SS:t@##############:WARNING: LoginBlocked DN (cn=vmware-applmgmtservice-########-####-####-####-############,cn=serviceprincipals,dc=vsphere,dc=local), error (9239)(Account access blocked)
YYYY-MM-DDTHH:MM:SS:t@###############:INFO: Bind failed () (9239)
YYYY-MM-DDTHH:MM:SS:t@###############:ERROR: VmDirSendLdapResult: Request (Bind), Error (LDAP_INVALID_CREDENTIALS(49)), Message (), (0) socket (127.0.0.1)
YYYY-MM-DDTHH:MM:SS:t@###############:ERROR: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "CN=vmware-applmgmtservice-########-####-####-####-############,cn=ServicePrincipals,dc=vsphere,dc=local", Method: SASL



Cause

The root cause is the expiration of the vmware-applmgmtservice service account password.

By default, this service account password expires every 90 days. Under normal conditions, the vCenter backup scheduler should automatically reset this password upon expiration. However, in affected versions, the scheduler fails to perform this reset, causing all subsequent scheduled backup attempts to fail due to "Invalid Credentials" and "Account access blocked" errors within the SSO and Directory service

 

Resolution

This issue is resolved in vCenter Server 8.0 Update 3h (Build 25092719). It is recommended to update to this version or higher to ensure the applmgmt service account password resets automatically as intended.

Workaround

If an immediate upgrade is not possible, the service account state can be cleared by restarting the Appliance Management Service:

Option 1: Using the VAMI (Web UI)

  1. Log in to the vCenter Appliance Management Interface: https://<vCenter-FQDN>:5480 as root.

  2. Click on the Services tab.

  3. Locate the VMware Appliance Management Service (applmgmt) and click Restart.

Option 2: Using the Command Line (SSH)

  1. Log in to the vCenter Server Appliance via SSH as root.

  2. Run the following command to restart the service:

    service-control --restart applmgmt

After the restart, the scheduled backup should trigger successfully at the next interval.

Additional Information

Release Notes Reference: The fix for this issue is documented in the vCenter Server 8.0 Update 3h Release Notes.

PR 3512033: When the VMware Appliance Management Service password expires, scheduled backups of vCenter fail.

Details: When the applmgmt service password expires, the password does not reset and scheduled backups of vCenter fail. This issue is resolved in this release. The fix ensures the system resets the applmgmt service password for scheduled backups upon expiration.

Powered by Wolken Software