Performance issues in the CA directory as a Session Store

Products

CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign-On SITEMINDER

Issue/Introduction

Performance issues are observed after deploying and enabling CA Directory as a Session Store in the environment – LDAP operations are slow.

Environment

Policy server R12.52 SP1 CR4 or below on Unix
Policy server R12.52 SP2 or below on windows
CA Directory R12 SP18 (multiple Data DSA, Router DSA)
Webserver and Agent version (various)

Cause

Default CA Directory Configuration:

SiteMinder Policy Server can get into a state where it is unable to keep up with Session Store maintenance.

Policy Server maintenance includes adding, updating, and deleting sessions.

If sessions get added faster than they are removed, over time this can cause CA Directory Session Store to fill up.

Use the following LDAP search to find out if the CA Directory Session Store is getting overwhelmed with session entries:

dxsearch -LLL -h <IPAddress:PortNumber> -b "<Session Store Root DN>" -D "<UserID>" -w <password> (SmSessionID=*) dxEntrycount

Resolution

Tune CA Directory Configuration (Changes need to apply on ALL Data DSAs):

The nature of the session data stored in the CA Directory Session Store is much different than that of a User Store or a Policy store.

The session data is very transient with a short life span.

Entries in the Session Store are added and then removed very quickly, in some cases within a few minutes.

Given the nature of this short-lived data, optimize some CA Directory configurations to provide better performance (1).

Limits:

max-op-size:

This command sets the maximum number of entries that a search or list can return.

In the size limits section of the smsession.dxc file, set the max-op-size setting to match the following example.

This value represents a high limit. This will limit the Policy Server to processing 1,000 objects per search query.

Modify Session Store .dxc file:DXHOME\config\limits\

set max-op-size = 1000;
multi-write queue:

The multi-write queue setting specifies the maximum number of transactions that are held in memory for a DSA that is unavailable.

Use this setting to support a server that is offline for a given period. The default value is 20,000.

Set the queue to handle from 300,000 through 500,000 transactions.

The optimal value allows a normal reboot of one of the Directory Server hosts without causing the multi-write queue to fill.

If the limit is hit for one queue (monitor this from the CA Directory alerts log), this DSA (currently marked MW-FAILED) will have all of its operations discarded (and it will be marked QUEUE-PURGED-OUT-OF-ORDER).

Modify session store .dxc file:DXHOME\config\limits\

set multi-write-queue = 300000;

Sizing Multi-write Queues (2).
multi-write outstanding ops limits:

This setting determines the number of updates that a DSA can send at one time, from its multi-write queue to a recovering DSA.

The default is 10.

Based on the number of potential updates in the multi-write queue, a DSA sending the default 10 updates at a time prolongs the automatic recovery of a DSA.

The recommended value is 1000.

Modify sessions store .dxc file:DXHOME\config\limits\

set multi-write-outstanding-ops = 1000;

Session store Initialization file:

Optimize the indexes of the data:

Limit the attributes indexed on the Directory by using the ‘set cache-index’ flag.

In the "# grid configuration" section – make sure these attributes are listed.

Modify session store .dxi file:DXHOME\config\servers\

NOTE: Verify that the cache-index attribute is above the lookup-cache attribute.

This is the list of needed fields to index:

set cache-index = smSessionId, smExpirationTime, smIdleExpirationTime, smSearchData, smVariableName, smFullVariableName;
set lookup-cache = true;

From a SiteMinder perspective:

If the Policy Server is configured to use variables: smSessionId, smExpirationTime, and smVariableName, then modify the dxi to state:

set cache-index = smSessionId, smExpirationTime, smVariableName

Save the file .dxi file.

Complete this on all Session Store Data DSAs.
multi-write-disp-recovery:

The default value is set to false.

This Session Store does not need this recovery method.

Modify sessions store .dxi file:DXHOME\config\servers\

set multi-write-disp-recovery = false;
In case CA Directory replication goes out of sync (Monitor Multi-write queue from the Directory alerts log, alarms are raised at 60%, 70%, 80%, 90% and 100% of queue size once reached), a manual recovery might be required.

Please refer to CA Directory documentation on how to handle manual recovery (3).
Disabled the Transaction log.

By default, each CA Directory DSA is configured to utilize a "transaction log".

Session Store does not need the Transaction log.

Modify sessions store .dxi file:DXHOME\config\servers\

NOTE: Add it below the "set multi-write-disp-recovery = false;" line.

# multiwrite DISP recovery

set multi-write-disp-recovery = false;
set disable-transaction-log = true;

Knowledge files:

Replication:

If Session Store DSAs are replicated, add the dsp-idle-time setting to the DSA knowledge.

Setting the idle time is useful so that the idle time is not too low.

If set too low, the link between a multi-write DSA and its recovering peer DSA can time out before the DSA is recovered.

Modify sessions store .dxc file:DXHOME\config\knowledge\

dsp-idle-time = 30
No need to back up the Directory.

There is no need to backup data that is always changing because a restore of any backup would be out-of-date and inaccurate.
Data Partitioning* (if necessary)

CA Directory supports data partitioning.

This will allow for the data (a single OU) to be divided up among multiple Data DSAs.

This can have a significant impact on performance by allowing more DSAs to handle the LDAP request load.

Data Partitioning will take advantage of all available hardware, whether extra CPUs on a single host or CPUs across multiple hosts.

More replicas would allow for more parallel writes to the Directory.

In the environments where additional servers are available or additional resources (CPUs or Cores) on a host are not being leveraged, then consider configuring CA Directory Data Partitioning.
SiteMinder Policy Server specific configurations:

The Policy Server uses a Connection pool model (default size 10):

A connection pool is a cache of backend repository (Database/LDAP) connections maintained so that the connections can be reused when future requests to the backend repository are required.

Connection pools are used to enhance the performance of executing commands on a backend repository.

Running with 50 threads that require a Session Store connection handle (default 10) and the LDAP operations takes time to complete processing.

This can cause a normal thread queue to build up.

What to look for:

Using Policy Server profiler logs, observe if a request has a delay before Lock LDAP handle – this would mean we need to increase the number of Session Store connections.

An example of what to look at in the profiler logs for this type of event:

Example:

2 seconds wait for thread ID (TID) 3003599728 before this worker thread was able to get an LDAP connection handle is shown below:

[13:23:38.671][26918][3003599728][Leave function CSm_Auth_Message::SetAuthContext][Sm_Auth_Message.cpp:4280][CSm_Auth_Message::SetAuthContext]
[13:23:40.671][26918][3003599728][Lock LDAP handle. slot=0 ld=0xb3071fe0][LdapStore.cpp:375][Lock_LdapHandle]

Resolution:

How to increase connections to the Session Store?

Modify the following registry key (NOTE: configure the value close to half the max worker thread value so you have at least one connection handle for every 2 worker threads. This recommendation would vary based on the hardware used and other environmental considerations. For optimal performance, we recommend testing these in a test environment and modify as needed to adjust to a given environment.)

HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\LdapSessionServer=240093784
MaxConnections= 0xa; REG_DWORD
SMDeleteSession (4)

SiteMinder Policy Servers configured to use CA Directory as a Session Store are responsible for creating sessions and then cleaning up the expired sessions.

The Policy Servers create sessions when users authenticate, then periodically, the Policy Server will search for sessions that have expired and remove them.

In high-volume environments, the SiteMinder Policy Servers tend to spend many more cycles creating/updating sessions than removing them.

Over some time, this will cause the CA Directory Session Store to fill with primarily expired sessions.

To counteract this issue, there's a specialized JAVA tool that can be configured as a cronjob or scheduled task, a tool called the SMDeleteSession.

This process remains running, all the while checking for expired sessions, then removing them.

Its only function is to remove expired sessions. (NOTE Policy Server session maintenance code will be modified in future releases making this tool not needed even for extremely high volume/traffic sites).

NOTE:

SMDeleteSession is not applicable for Policy Servers at or above SiteMinder 12.52 SP1 CR5 on the SP1 stream. For 12.52 SP2 this tool may still be needed.

DISCLAIMER:

CA IS DELIVERING THE FILES "AS IS" AND CA MAKES NO WARRANTIES CONCERNING THE FILES, EXPRESS, IMPLIED, OR ARISING BY CUSTOM OR TRADE USAGE, AND SPECIFICALLY MAKES NO WARRANTY OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR PURPOSE. CA's EXPRESS WARRANTIES SHALL NOT BE ENLARGED, DIMINISHED, OR AFFECTED BY, AND NO OBLIGATION OR LIABILITY SHALL ARISE OUT OF, CA's RENDERING OF TECHNICAL OR OTHER ADVICE OR SERVICE IN CONNECTION WITH THE FILES.

IN NO EVENT SHALL CA BE LIABLE FOR ANY LOSS OR DAMAGE OF ANY KIND, INCLUDING BUT NOT LIMITED TO, SPECIAL, INDIRECT, PUNITIVE INCIDENTAL, OR CONSEQUENTIAL DAMAGES, ARISING OUT OF THE DELIVERY, USE, EVALUATION, SUPPORT, OR OPERATION OF THE FILES PROVIDED EVEN IF CA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.