AD Userstore using LDAP namespace is returning "Search Operation Failed: timelimit exceeded" and does not show the Userstore Contents.

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• CA Single Sign-On

Issue/Introduction

Description:

Customer had v5.5SP2 Policy Server and Active Directory as their user-store.
They have many objects in the AD(100s of OUs, 1000s of Groups and its members).
As it is v5.5SP2, they use LDAP namespace for this AD user-store(AD namespace is introduced in v6).

When they click on "View Contents" button from the User-Store object in the Policy Server Administration UI, it displays the user-store contents in few seconds.

ISSUE: When they installed v6.0SP5 policy server on a new machine and pointed to the 5.x policy store, the "View Contents" return after 5 minutes and throws "Search Operation Failed: timelimit exceeded". It does not show the user-store contents. However, users are able to get authenticate and authorized.

Solution:

IMPORTANT: This article contains information about modifying the registry.
Before you modify the registry, make sure to create back up of the registry and ensure that you understand how to restore the registry if a problem may occur.
For more information about how to back up, restore, and edit the registry, please review the relevant Microsoft Knowledge Base articles on support.microsoft.com.

This is an expected behavior if they installed their Policy Server on a new machine.

The key point to this issue is the features enabled in the registry.
This registry in question is "EnableObjectCategory".
By default, if you click on "View Contents" button from the user-store object, policy server will perform the following 5 LDAP searches.
(objectclass=Organization)
(objectclass=OrganizationalUnit)
(objectclass=groupOfUniqueNames)
(objectclass=groupOfNames)
(objectclass=group)

If you have many "group" objects with many "members", this causes all requests to this AD to be slow.
Search for (objectclass=Organization) would also reach search timelimit(default value : 30 seconds) regardless of whether you have Organization object or not.
The same goes to ping search(objectclass=*).

Policy Server will send the original (objectclass=Organization) query to the AD and it times out.
Policy Server determines that it should try to send the request again and it timeout again.
So, each objectclass will take 1 minute and as there are 5 objectclasses, it will take 5 minutes to get the error message.

This "objectclass" is not indexed so indexing the AD for "O", "OU", "objectclass" and "member" will help the situation but there is a better way to achieve this.

HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\Ds\LDAPProvider
EnableObjectCategory = 1(DWORD)

Above registry EnableObjectCategory=1 will change the LDAP query from objectclass to objectCategory if the LDAP is determined to be AD.
Policy Server will perform the 5 ldap searches using objectCategory instead of objectclass.

ActiveDirectory natively indexes this objectCategory (http://msdn.microsoft.com/en-us/library/ms677612(VS.85).aspx) so the "View Contents" would be displayed in few seconds.

However, this EnableObjectCategory is set by default to "0" which means disabled.

Customer needs to ensure that all the registry-enabled features are recorded down and added in to the upgrade plans to maintain the same behavior.

Environment

Release:
Component: SMPLC