Investigating high Max. Response Time, with BCAAA Authentication

Products

• ASG-S500

Issue/Introduction

Investigating high Max. Response Time, with BCAAA Authentication

Resolution

Having investigated the uploaded logs, we see tons of authentication failures. Please see the attached .csv file, for reference. Investigating further, we see very high Max. Resp. time, for the "auth.qa" authentication realm/server. This response time relates to the authentication traffic between the Proxy and the BCAAA server(s). Please see the snippet below, for reference.

For more on the authentication statistics, please refer to the snippets below.

The possible reasons for the high response time, with BCAAA are:

• Overloaded DC and response is getting delayed.

• BCAAA is using NetLogon which is connected to a slow or remote DC

• Too many authentication attempts from proxy and requests are getting queued.

Recommendations for managing/resolving the high response time:

• To reduce the authentication attempts from the proxy side, you can use Surrogate modes such as "Proxy-IP". We see in the logs that you are utilizing this authentication mode.

• Use the Nltest utility to confirm and point BCAAA server to netlogon to a closer DC. Ref: https://knowledge.broadcom.com/external/article?legacyId=TECH241427

• Increase SChannel by increaxing MaxConCurrentAPI. https://knowledge.broadcom.com/external/article?legacyId=TECH246270

• Move to Kerberos

Too many authentication attempt: To further check to see if something like that is happening or not you may check the eventlog on the proxy using the url: https://x.x.x.x:8082/eventlog/fetch=0xffffffff

Under eventlog  there will be many lines within a short time frame (i.e multiple times per second) may show up like below

Authentication failed from x.x.x.x: user 'abcd' (realm IWA)". We see these in the eventlogs already. This further checks would only further validate what's already known.

Note:

For the too many authentication attempts , chances are we are overloading the BCAAA and / or DC unnecessarily. The best remedy for this is to apply authentication best practices. Please see the attached.

Authentication best practices CPL policy is built with known source NTLM user agents which fails to perform proxy authentication and creates a loop of failed authentication attempts.  Note that these authentication failure messages are very common for any deployment which has IWA authentication in place. We want to limit their number with the help authentication best practices CPL. Also, a number of finetuning can be done on top of existing best practices CPL.

For any BCAAA-specific failures, the BCAAA application log, from the Windows server will be helpful. Therein, the event IDs returned, for the failures, should be checked and matched with the descriptions provided in the Tech. Article with the URL below.

https://knowledge.broadcom.com/external/article/165386/bcaaa-event-id-explanations.html