Customer reported unexpected authentication outages occurring mostly at night when little load would have been expected on the BCAAA server Customer was using Kerberos with some residual NTLM authentication. BCAAA 6.1 installed on MS Windows 2008 servers, memory and CPU utilization where low. Restarting the BCAAA service would get authentication services working again.
While the customer was using Kerberos there were still some NTLM authentication request seen in the logs, this is not unusual as Windows will downgrade to NTLM for any number of reasons, such as a user logging into the local domain (local workstation login). In this particular case a small number of NTLM requests where sequestering the available numthreads and causing other authentication requests to build up
The default value for numthreads in BCAAA.INI is 2, this value can be increased to 64 as long as the BCAAA servers has sufficient memory to cope with the extra workload (Blue Coat have tested as high as 150) . It is important the the Domain Controllers be running windows 2008 R2 or higher.