Symantec Directory : Gradual increase in response time when performing search with SPR control
Article ID: 246297
Updated On:
Products
Issue/Introduction
You may experience a performance related issue when searching a very large data set along with use of SPR (Simple Paged Results) control.
For an example, you expect to return 70 million objects from a DIT (Directory Information Tree.. aka OU) with SPR set to 1000. Meaning return 1000 objects at a time in small batches until all 70 million objects are retrieved.
Doing so, you will see that the initial search results (1000 objects at a time) are progressing within a reasonable SLO (Service Level Objective) but as it progresses further, the time taken to return each batch of 1000 object gets longer and longer.
If you have query log set, you will see something similar to this:
[6] 20220718.155916.060 100.1 BIND <IP address> dn="cn=myid,dc=mycompany,dc=com" source="client"
[0] 20220718.155916.061 100.1 RESULT success
[5] 20220718.155916.220 100.5 SEARCH dn="ou=Users,dc=mycompany,dc=com" scope=subtree filter=(objectClass=myobjectclass) eis=7 source="client"
[6] 20220718.155916.245 100.5 RESULT success 1000 entries 25 msecs
[3] 20220718.155921.375 96.6 SEARCH dn="ou=Users,dc=mycompany,dc=com" scope=subtree filter=(objectClass=myobjectclass) eis=7 source="client"
[4] 20220718.155921.400 96.6 RESULT success 1000 entries 25 msecs
[2] 20220718.155922.849 95.6 SEARCH dn="ou=Users,dc=mycompany,dc=com" scope=subtree filter=(objectClass=myobjectclass) eis=7 source="client"
[6] 20220718.155922.869 95.6 RESULT success 1000 entries 20 msecs
[2] 20220718.155927.256 98.6 SEARCH dn="ou=Users,dc=mycompany,dc=com" scope=subtree filter=(objectClass=myobjectclass) eis=7 source="client"
[4] 20220718.155927.280 98.6 RESULT success 1000 entries 24 msecs
....
....
....
[7] 20220718.204127.656 94.893 SEARCH dn="ou=Users,dc=mycompany,dc=com" scope=subtree filter=(objectClass=myobjectclass) eis=7 source="client"
[4] 20220718.204131.994 94.893 RESULT success 1000 entries 4338 msecs
[4] 20220718.204136.921 98.731 SEARCH dn="ou=Users,dc=mycompany,dc=com" scope=subtree filter=(objectClass=myobjectclass) eis=7 source="client"
[6] 20220718.204141.602 98.731 RESULT success 1000 entries 4681 msecs
As you can see above, initially it was taking ~20 milliseconds and as time went by it was taking ~5000 milliseconds (i.e. 5 seconds)
Environment
Release : 14.1
Component : CA Directory
Resolution
This is working as design when it comes to SPR LDAP control. Here is why you are seeing what you seeing.
DXserver does not store any search results in cache. So for every search request, DXserver need to get the requested entries based on the filter and for SPR search, DXserver also needs to get the requested entries based on the filter, but should only return the entries based on page number and page size.
For example:
If DXserver (aka DSA) has 70,000,000 entries which match the filter (say cn=a*) and when we perform SPR search with page size set to 1000:
** With 1st page, DSA gets 1000 entries and returns 1000 entries (1-1000 entries)
** With 2nd page, DSA gets 2000 entries and returns last 1000 entries (1001-2000 entries)
** With 3rd page, DSA gets 3000 entries and returns last 1000 entries (2001-3000 entries)
and so on...
** With last "-1" page, DSA gets 69,999,900 entries and returns last 1000 entries (69,998,901-69,999,900 entries)
** With last page, DSA gets 70,000,000 entries and return last 1000 entries (69,999,901-70,000,000 entries)
Since the DSA has to fetch the needed entries, as page number increases, the number of fetched entries are also increased though we return only page size output entries.
We do not see this issue when the number of entries matching the filter are small.
This is expected behavior. Due to this, you see performance getting degraded over time as you are trying to fetch ~70 million entries.
Feedback
