Scheduled file-based backup fails to trigger/start on vCenter server

Products

• VMware vCenter Server

Issue/Introduction

• Schedule file-based backup job fails to start or trigger on vCenter server.
• Manual file-based backup works fine on vCenter server.
• Trying to configure or edit the VAMI backup returns error: "Path not exported by the remote filesystem."

 

/var/log/vmware/applmgmt/backupScheduler.log

YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-######] [VapiClientHelper::get_saml_token_with_svc_user:VapiClientHelper.py:117] ERROR: Failed to get HOK token with error SoapException:
faultcode: ns0:FailedAuthentication
faultstring: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}
faultxml: <?xml version='1.0' encoding='###-#'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}</faultstring></S:Fault></S:Body></S:Envelope>.
YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-######] [VapiClientHelper::get_vapi_stub_with_saml_auth:VapiClientHelper.py:92] ERROR: Failed to getting vapi stub with svc user: name 'svcUtil' is not defined
YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-#######] [VapiClientHelper::get_saml_token_with_svc_user:VapiClientHelper.py:117] ERROR: Failed to get HOK token with error SoapException:
faultcode: ns0:FailedAuthentication
faultstring: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}
faultxml: <?xml version='1.0' encoding='###-#'?><S:Envelope xmlns:S="http://schemas.xmlsoap.org/soap/envelope/"><S:Body><S:Fault xmlns:ns4="http://www.w3.org/2003/05/soap-envelope"><faultcode xmlns:ns0="http://docs.oasis-open.org/ws-sx/ws-trust/200512">ns0:FailedAuthentication</faultcode><faultstring>Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}</faultstring></S:Fault></S:Body></S:Envelope>.
YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-#######] [VapiClientHelper::get_vapi_stub_with_saml_auth:VapiClientHelper.py:92] ERROR: Failed to getting vapi stub with svc user: name 'svcUtil' is not defined
...
YYYY-MM-DDTHH:MM:SS [0] [MainProcess:PID-#######] [Scheduler::ExecScheduleRun:Scheduler.py:137] ERROR: Failed to issue the Schedules.run request. Exception: {messages : [LocalizableMessage(id='com.vmware.applmgmt.backup.plugin.fs_path_not_found', default_message='Path not exported by the remote filesystem.', args=['Plugin error occurred. ErrCode: 151, Args: ()'], params=None, localized=None)], data : None, error_type : None}
Traceback (most recent call last):
  File "/usr/lib/applmgmt/backup_restore/py/vmware/appliance/backup_restore/Scheduler.py", line 133, in ExecScheduleRun
    status = svc_handle.run(scheduleId, comment='SCHEDULED')
  File "/usr/lib/applmgmt/pyclient/applmgmt_client-1.0-py2.7.egg/com/vmware/appliance/recovery/backup_client.py", line 1197, in run
    return self._invoke('run',
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke
    return self._api_interface.native_invoke(ctx, _method_name, kwargs)
  File "/usr/lib/applmgmt/vapi/lib/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 295, in native_invoke
    raise TypeConverter.convert_to_python(method_result.error,  # pylint: disable=E0702
com.vmware.vapi.std.errors_client.Error: {messages : [LocalizableMessage(id='com.vmware.applmgmt.backup.plugin.fs_path_not_found', default_message='Path not exported by the remote filesystem.', args=['Plugin error occurred. ErrCode: 151, Args: ()'], params=None, localized=None)], data : None, error_type : None}


 

/var/log/vmware/sso/vmware-identity-sts.log:

YYYY-MM-DDTHH:MM:SS WARN sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.interop.ldap.LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.OpenLdapClientLibrary, error code: 49

YYYY-MM-DDTHH:MM:SS WARN sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.ServerUtils] cannot bind connection: [ldap://VCENTER.FQDN:389, vmware-applmgmtservice-########-####-####-####-############@vsphere.local]

YYYY-MM-DDTHH:MM:SS ERROR sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.ServerUtils] cannot establish ldap connection with URI: [ldap://VCENTER.FQDN:389] because [Invalid credentials] therefore will not attempt to use any secondary URIs

YYYY-MM-DDTHH:MM:SS WARN sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider] Failed to authenticate using SRP binding

com.vmware.identity.interop.ldap.InvalidCredentialsLdapException: Invalid credentials

at com.vmware.identity.interop.ldap.LdapErrorChecker$28.RaiseLdapError(LdapErrorChecker.java:415) ~[libvmware-identity-platform.jar:?]

at com.vmware.identity.interop.ldap.LdapErrorChecker.CheckError(LdapErrorChecker.java:1102) ~[libvmware-identity-platform.jar:?]

at com.vmware.identity.interop.ldap.OpenLdapClientLibrary.CheckError(OpenLdapClientLibrary.java:1282) ~[libvmware-identity-platform.jar:?]

at com.vmware.identity.interop.ldap.OpenLdapClientLibrary.CheckError(OpenLdapClientLibrary.java:1275) ~[libvmware-identity-platform.jar:?]

...

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-util.jar:9.0.86]

at java.lang.Thread.run(Thread.java:750) [?:1.8.0_401]

YYYY-MM-DDTHH:MM:SS WARN sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider] The user is ###-####### and failed to authenticate.

YYYY-MM-DDTHH:MM:SS ERROR sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.IdentityManager] Failed to authenticate principal [vmware-applmgmtservice-########-####-####-####-############@vsphere.local] for tenant [vsphere.local]

javax.security.auth.login.LoginException: Login failed

at com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider.authenticate(VMwareDirectoryProvider.java:428) ~[libvmware-identity-idm-server.jar:?]

at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:3134) [libvmware-identity-idm-server.jar:?]

at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:10530) [libvmware-identity-idm-server.jar:?]

...

at com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider.getConnection(VMwareDirectoryProvider.java:6700) ~[libvmware-identity-idm-server.jar:?]

at com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider.authenticate(VMwareDirectoryProvider.java:400) ~[libvmware-identity-idm-server.jar:?]

... 69 more

 

YYYY-MM-DDTHH:MM:SS INFO sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.diagnostics.VmEventAppender] EventLog: source=[VMware Identity Server], tenant=[vsphere.local], eventid=[USER_NAME_PWD_AUTH_FAILED], level=[ERROR], category=[VMEVENT_CATEGORY_STS], text=[ParameterizedMessage[messagePattern=Failed to authenticate principal [{}]. User password expired., stringArgs=[vmware-applmgmtservice-########-####-####-####-############@vsphere.local], throwable=null]], detailText=[null], corelationId=[########-####-####-####-############], timestamp=[##########]

YYYY-MM-DDTHH:MM:SS ERROR sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.IdentityManager] Failed to authenticate principal [vmware-applmgmtservice-########-####-####-####-############@vsphere.local]. User password expired.

YYYY-MM-DDTHH:MM:SS INFO sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.IdentityManager] Authentication failed for user [vmware-applmgmtservice-########-####-####-####-############@vsphere.local] in tenant [vsphere.local] in [20] milliseconds with provider [vsphere.local] of type [com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider]

YYYY-MM-DDTHH:MM:SS ERROR sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.idm.server.ServerUtils] Exception 'com.vmware.identity.idm.PasswordExpiredException: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}'

com.vmware.identity.idm.PasswordExpiredException: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}

at com.vmware.identity.idm.server.provider.vmwdirectory.VMwareDirectoryProvider.checkUserAccountFlags(VMwareDirectoryProvider.java:1458) ~[libvmware-identity-idm-server.jar:?]

at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:3159) ~[libvmware-identity-idm-server.jar:?]

at com.vmware.identity.idm.server.IdentityManager.authenticate(IdentityManager.java:10530) [libvmware-identity-idm-server.jar:?]

at com.vmware.identity.idm.client.CasIdmClient.authenticate(CasIdmClient.java:1303) [libvmware-identity-idm-client.jar:?]

at com.vmware.identity.sts.idm.impl.AuthenticatorImpl.authenticate(AuthenticatorImpl.java:91) [libsts.jar:?]

...

at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-util.jar:9.0.86]

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) [tomcat-util.jar:9.0.86]

at java.lang.Thread.run(Thread.java:750) [?:1.8.0_401]

YYYY-MM-DDTHH:MM:SS INFO sts[84:tomcat-http--46] [CorId=########-####-####-####-############] [com.vmware.identity.sts.ws.SOAPFaultHandler] Returning a SOAP Fault with code: ns0:FailedAuthentication and description: Password of the user logging on is expired. :: Password of the user logging on is expired. :: User account expired: {Name: vmware-applmgmtservice-########-####-####-####-############, Domain: vsphere.local}

 

 

/var/log/vmware/vmdird/vmdird.log:

YYYY-MM-DDTHH:MM:SS:t@##############:WARNING: LoginBlocked DN (cn=vmware-applmgmtservice-########-####-####-####-############,cn=serviceprincipals,dc=vsphere,dc=local), error (9239)(Account access blocked)

YYYY-MM-DDTHH:MM:SS:t@###############:INFO: Bind failed () (9239)

YYYY-MM-DDTHH:MM:SS:t@###############:ERROR: VmDirSendLdapResult: Request (Bind), Error (LDAP_INVALID_CREDENTIALS(49)), Message (), (0) socket (127.0.0.1)

YYYY-MM-DDTHH:MM:SS:t@###############:ERROR: Bind Request Failed (127.0.0.1) error 49: Protocol version: 3, Bind DN: "CN=vmware-applmgmtservice-########-####-####-####-############,cn=ServicePrincipals,dc=vsphere,dc=local", Method: SASL

 

 

Environment

vCenter Server Appliance 8.0

Cause

Triggering a scheduled backup requires authentication with service accounts. Service account password expires in 90 days.

Once the password is expired, the backup scheduler failed to reset the service account password.

 

Resolution

This is currently a known issue and Broadcom Engineering team is working on a fix.

If you encounter the issue, please contact Broadcom support.

Workaround: 

1. Login to Appliance management on vCenter https://<vCenter FQDN>:5480 as root
2. Click on services Tab - Restart VMware appliance management service

or

1. SSH to vCenter appliance as root 
2. Restart applmgmt service using the command service-control --restart applmgmt