CLI Authentication failing on multiple CLI calls with 401 Error
Article ID: 268677
Updated On:
Products
Issue/Introduction
A client is accessing CA PAM CLI simultaneously from multiple machines using the same username and password.
For instance
....
Authentication error is obtained from time to time both as a result and in the Password Authority tomcat log:
CLI command "searchAgent" failed with error "PAM-CM-0567: Failed to authenticate with the Password Authority service.".
and in the tomcat log:
INFO: SecurityContextFactory.create(map) Unable to create login context: GK, error code: 12, error msg: GKLoginModule.login User 'UserImpl[ID=17423001]{
User=testProblem
Password=********,
FirstName=Mytest,
LastName=Mytest,
LdapDN=null,
AuthenticationType=CSPM,
ViewType=admin,
GkUserID=15928001,
ServerKeyID=-1,
UserGroupIDs=[],
Status=ACTIVE,
LastLogin=null,
FailedLoginAttempts=0,
Attributes=NULL,
CreateUser=super (2021-02-03 08:56:22.000), UpdateUser=super (2023-02-24 16:10:01.000), Hash=VtXxJf9sHN+w26KxWcy26CchQSQ=
}' authentication failed via verifyUserCredentials in serviceController.php
Environment
CA PAM 4.0.X and likely also lower versions
Cause
When the CLI command is launched, the admin username and password need to be accessed. While they are being retrieved the the login_history table must be updated to reflect the the fact that the admin is logging in into PAM to perform the command. This effectively locks the table row affected while the it is being updated. The time spent with the lock is very short and, in general, no lock should be expected for other or the same user accessing the same table concurrently.
However, if enough accesses using the same username are done concurrently it is a matter of time while two CLI calls try to update the same row of the login_history table simultaneously, and this behaviour will be observed
It is important to remark that this is not a problem caused by PAM but the regular behaviour of MySQL. The same issue can be simulated by just creating a table with a single row and programmatically accessing it.
CA PAM 4.1.2 introduces a possible mechanism for minimizing these use cases: if the first access attempt fails, swallow the exception and proceed with a second attempt, then a third attempt until giving up. This is what is happening in this case: CA PAM tries to access the login_history for two concurrent calls, it fails, tries a second time and it usually works. It is only from time to time that a call fails because it goes over the three retries.
Resolution
There are several possible approaches to resolving or minimising this problems
Programmatically increase again the number of retries, for instance to 5 or 8. The problem with this is that it may be harder to happen, but the same use case will eventually be observed. It is just a matter of time
Use different users for performing the same command for different groups, so that there are no locks at row level in login_history.
Take the authentication out of the loop. The CLI commands are meant to be for admin purposes, so they are stateless. This means one does the command, but the admin user does not remain authenticated. At the next command, username and password need to be specified, etc, In REST API calls, for instance, the username and password need to be provided just one, at the beginning of the session, and the rest of the commands may be run without having to reauthenticate. This greatly reduces or even suppresses the possibility of simultaneous access to the table causing an error to appear
Feedback
