We had temporary failures getting AD accounts updated using a service account that is configured with rights to update the passwords of the managed accounts, as well as unlock them. There is no useful information in the tomcat log at any log level. The update attempt fails with an authentication error, code 1604. The service account itself has a valid password and authenticated successfully, but could not update the password of the managed account because of a problem on the AD side. The tomcat logs at Info log level did not show any error from the service account, only the failed authentication attempts by the managed account.
Here is an example of two consecutive log lines, the first one reporting successful authentication by the service account, the next one already belonging to the next login attempt by the managed account:
May 29, 2020 3:19:05 AM com.cloakware.cspm.server.plugin.targetmanager.WindowsDomainServiceTargetManager loginToActiveDirectoryServer
INFO: Successfully authenticated to Active Directory and set the last known good host to 'lvntest000536.bpc.broadcom.net'
May 29, 2020 3:19:05 AM com.cloakware.cspm.server.plugin.targetmanager.WindowsDomainServiceTargetManager updatePasswordInActiveDirectory
INFO: Retrieve the DN from the Active Directory for user, changemypassword,CN=Changemy Password,CN=Users,DC=rppam,DC=net
Between these two log lines the service account ran into an error making a call into AD, but this error is not seen in the tomcat logs nor shown on the UI, hindering determination of root cause and resolution of the problem.
The workflow for a password update for an Active Directory target account (managed account) by a service account is as follows:
1) PAM tries to logon to AD using the DN of the managed account and the new password - This will fail
2) PAM tries to logon to AD using the UPN of the managed account and the new password - This will fail
3) PAM logs on as the service account, unlocks the managed account if needed and sets the new password.
4) PAM tries to logon to AD using the DN of the managed account and the new password - This will be successful, if 3) was successful, but fail otherwise
5) If 4) failed, PAM tries to logon to AD using the UPN of the managed account and the new password - This will fail
Steps 1 and 2 are useful for the case where a PAM admin enters a password on the UI, because that typically is the correct current password. However, this is done for any type of password update, even when a scheduled job generates a new password, in which case those attempts are bound to fail. This is the current workflow in PAM that may change in future releases.
The main flaw with regard to the problem discussed here is that any error during step 3 is caught without logging or otherwise saving the error. PAM just proceeds to step 4 This is not always wrong as it is possible that an error is returned, such as a timeout error, even though the password of the managed account in fact was or is being changed successfully. But not giving the PAM admin any information on what error was encountered obscures the problem and delays resolution. Since PAM just continues with steps 4 an 5, the last error encountered will be an authentication error and that is what will be reported on the UI. In the Account Passwords Update Attempts report the failed password update will show error code 1604.
Release : 3.3
Component : PRIVILEGED ACCESS MANAGEMENT
For now PAM Engineering is not changing the program flow, but code is being added to log errors coming back from AD in the PAM tomcat log (catalina.out), which can be downloaded from the Configuration > Diagnostics > Diagnostic Logs > Download page. This is scheduled to be included in PAM 3.3.4, 3.4.2 and 3.5+ and should help identify the problem faster.
If you see the log pattern discussed above, check the AD event logs for any errors right after the successful service account logon.