Slow LDAP query after load balancer
Article ID: 43337
Updated On:
Products
Issue/Introduction
Issue:
Clients reporting slowness in application access
Environment:
Windows policy server Version: 12.5; Update: 00.03; Build: 875; CR: 03;
Analysis: Policy server profiler logs, they show transactions taking time to process
TIME/Number of TX
1 sec 500 ms - 1 sec 600 ms 1715
1 sec 600 ms - 1 sec 700 ms 1315
1 sec 700 ms - 1 sec 800 ms 1399
1 sec 800 ms - 1 sec 900 ms 1013
1 sec 900 ms - 2 secs 981
2 secs - 3 secs 6020
3 secs - 4 secs 2522
4 secs - 5 secs 1089
5 secs - 6 secs 417
6 secs - 7 secs 170
7 secs - 8 secs 62
8 secs - 9 secs 30
9 secs - 10 secs 4
10 secs - 11 secs 1
11 secs - 12 secs 2
Break down of where time spent, each transaction performs multiple LDAP search, each search can take anywhere from anywhere from: 500ms to 5 seconds
LDAP Search/Number of Transactions
900 ms - 1 sec 4584
1 sec - 1 sec 100 ms 4385
1 sec 100 ms - 1 sec 200 ms 3077
1 sec 200 ms - 1 sec 300 ms 3046
1 sec 300 ms - 1 sec 400 ms 1935
1 sec 400 ms - 1 sec 500 ms 1465
1 sec 500 ms - 1 sec 600 ms 1371
1 sec 600 ms - 1 sec 700 ms 1127
1 sec 700 ms - 1 sec 800 ms 1015
1 sec 800 ms - 1 sec 900 ms 566
1 sec 900 ms - 2 secs 596
2 secs - 3 secs 2101
3 secs - 4 secs 288
4 secs - 5 secs 73
We do not see waiting for the handle during transaction processing, policy server sends multiple searches over the handle at the same time, however the examples below looks like we are buffering the results before continuing the transaction - this set of transactions all take the exact same amount of time, four different threads using the same handle - results are buffered
[2256][5312][05/09/2016][15:54:07.129][CSmDsLdapConn::SearchExts][SmDsLdapConnMgr.cpp:1205][LDAP search of uid=* took 1 seconds and 203156 microseconds]
[2256][6096][05/09/2016][15:54:07.129][CSmDsLdapConn::SearchExts][SmDsLdapConnMgr.cpp:1205][LDAP search of uid=* took 1 seconds and 203156 microseconds]
[2256][5952][05/09/2016][15:54:07.129][CSmDsLdapConn::SearchExts][SmDsLdapConnMgr.cpp:1205][LDAP search of uid=* took 1 seconds and 203156 microseconds]
[2256][5964][05/09/2016][15:54:07.129][CSmDsLdapConn::SearchExts][SmDsLdapConnMgr.cpp:1205][LDAP search of uid=* took 1 seconds and 203156 microseconds
Environment
Component:
Resolution
Resolution:
In logs provided we only see one LDAP bank to process requests from 20 threads.
Need to expand the bandwidth for Policy Server processing LDAP request to more than one BANK (load balance)
Information on how to create addition independent connections to the user store – this will allow each bank to failover one at a time
Unlike ODBC connection model (connection pooling) LDAP implementation is one LDAP bank using one connection to search and another to BIND.
Ideation you can vote on the add connections pooling for LDAP user store: https://communities.ca.com/ideas/235718429
To handle load the User Directory object can be load balanced, which creates multiple LDAP Banks to handle the incoming load:
If there is only one physical LDAP server the need to create multiple DNS names or HOST entries the policy server local host file may be needed to handle load – IDEALLY a future release of SiteMinder will do connection pooling to the LDAP server.
Below configuration creates 6 LDAP Banks to the same hosts (Primary 10.130.158.108 Failover 10.130.158.109)
NOTE: If the same HOSTNAME or IP address is repeated 6 times SiteMinder will mark all connections bad to the same name if error occurs on a single request (A B, C D, E F, G H, I J, K L)
SiterMInder UserDir Object: CA_Dir001andDir002_LB:
CAldap001-1.example.com:2001 CAldap002-1.example.com:2001,
CAldap001-2.example.com:2001 CAldap002-2.example.com:2001,
CAldap001-3.example.com:2001 CAldap002-3.example.com:2001,
CAldap001-4.example.com:2001 CAldap002-4.example.com:2001
CAldap001-5.example.com:2001 CAldap002-5.example.com:2001
CAldap001-6.example.com:2001 CAldap002-6.example.com:2001
/etc/hosts OR C:\Windows\System32\drivers\etc\hosts
10.0.0.1 CAldap001-1.example.com
10.0.0.1 CAldap001-2.example.com
10.0.0.1 CAldap001-3.example.com
10.0.0.1 CAldap001-4.example.com
10.0.0.1 CAldap001-5.example.com
10.0.0.1 CAldap001-6.example.com
10.0.0.1 CAldap002-1.example.com
10.0.0.1 CAldap002-2.example.com
10.0.0.1 CAldap002-3.example.com
10.0.0.1 CAldap002-4.example.com
10.0.0.1 CAldap002-5.example.com
10.0.0.1 CAldap002-6.example.com
Feedback
