Performance issues observed after deploying/enabling CA directory as a session store in the environment

book

Article ID: 40273

calendar_today

Updated On:

Products

CA Single Sign On Secure Proxy Server (SiteMinder) AXIOMATICS POLICY SERVER CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign-On

Issue/Introduction

PROBLEM:

 Performance issues observed after deploying/enabling CA directory as a session store in the environment – LDAP operations are slow.

ENVIRONMENT

  • Policy server R12.52 SP1 CR4 or below on Unix
  • Policy server R12.52 SP2 or below on windows
  • CA Directory R12 SP18 (multiple Data DSA, Router DSA)
  • Webserver and Agent version (various)

 

CAUSE:

 

Default CA Directory Configuration: Single sign-on policy server can get into a state where it is unable to keep up with Session store maintenance.  Policy server maintenance includes adding/updating/deleting sessions. If sessions get added faster than they are removed, over time this can cause CA Directory to fill up.
You can use the following LDAP search to find out if your directory is getting overwhelmed with session entries:

 

dxsearch -LLL -h <IPAddress:PortNumber> -b "<Session Store Root DN>" -D "<UserID>" -w <password> (SmSessionID=*) dxEntrycount

 

RESOLUTION:

Tune CA Directory Configuration: (Changes need to apply on ALL Data DSAs)

The nature of the session data stored in the Directory is much different than that of a User Store or a Policy store.  The session data is very transient with a short life span.  Entries in the session store are added and then removed very quickly, in some cases within a few minutes.  Given the nature of this short lived data, we are able to optimize some CA Directory configurations to provide better performance.

Limits:

max-op-size: This command sets the maximum number of entries that a search or list can return.  In the size limits section of the smsession.dxc file, set the max-op-size setting to match the following example. This value represents a high limit. This will limit the policy server to process 1,000 objects per search query.  Modify session store .dxc file:DXHOME\config\limits\

set max-op-size = 1000;

multi-write queue: The multi-write queue setting specifies the maximum number of transactions that are held in memory for a DSA that is unavailable. Use this setting to support a server that is offline for a given period. The default value is 20,000. Set the queue to handle from 300,000 through 500,000 transactions. The optimal value allows a normal reboot of one of the directory server hosts without causing the multi-write queue to fill. If the limit is hit for one queue (you can monitor this from the CA Directory alerts log), this DSA (currently marked MW-FAILED) will have all of its operations discarded (and it will be marked QUEUE-PURGED-OUT-OF-ORDER).  Modify session store .dxc file:DXHOME\config\limits\

set multi-write-queue = 300000;

Sizing Multi-write Queues see TechNote: tec408748

multi-write outstanding ops limits: This setting determines the number of updates that a DSA can send at one time, from its multi-write queue to a recovering DSA. The default is 10. Based on the number of potential updates in the multi-write queue, a DSA sending the default 10 updates at a time prolongs the automatic recovery of a DSA.  The recommenced value is 1000 Modify sessions store .dxc file:DXHOME\config\limits\

set multi-write-outstanding-ops = 1000;

Session store Initialization file:

Optimize the indexes of the data:  Limit the attributes indexed on the directory by using the ‘set cache-index’ flag.  In the “# grid configuration” section – make sure these attributes are listed. Modify session store .dxi file:DXHOME\config\servers\

NOTE: Verify that the cache-index attribute is above the lookup-cache attribute

These are the list of fields that we need to index:

set cache-index = smSessionId, smExpirationTime, smIdleExpirationTime, smSearchData, smVariableName, smFullVariableName;
set lookup-cache = true;

From a SiteMinder perspective:

If you have configured your SM Policy Server to use variables: smSessionId, smExpirationTime, smVariableName, 

then modify the dxi to state:

set-cache-index = smSessionId, smExpirationTime, smVariableName 

 Save the file .dxi file.  Complete this on all session store Data DSAs

multi-write-disp-recovery: The default value is set to false. This session store does not need this recovery method. Modify sessions store .dxi file:DXHOME\config\servers\

set multi-write-disp-recovery = false;

In case CA Directory replication goes out of sync (Monitor Multi-write queue from the directory alerts log, alarms are raised at 60%, 70%, 80%, 90% and 100% of queue size once reached), a manual recovery might be required. Please refer to CA Directory documentation on how to handle manual recovery (TechNote: tec408747).

Disabled the Transaction log. By default, each CA Directory DSA is configured to utilize a ‘transaction log’.  Session store has no need for the Transaction log. Modify sessions store .dxi file:DXHOME\config\servers\

NOTE:  Add it below the ‘set multi-write-disp-recovery = false;’ line.

# multiwrite DISP recovery
set multi-write-disp-recovery = false;
set disable-transaction-log = true;

Knowledge files:

Replication: If session store DSAs are replicated, add the dsp-idle-time setting to the DSA knowledge. Setting the idle time is useful so that the idle time is not too low. If set too low, the link between a multi-write DSA and its recovering peer DSA can time out before the DSA is recovered. Modify sessions store .dxc file:DXHOME\config\knowledge\

dsp-idle-time = 30

No need to back-up the directory

There is no need to backup data that is always changing because a restore of any backup would be out-of-date and inaccurate.

Data Partitioning* (if necessary)

CA Directory supports data partitioning.  This will allow for the data (a single OU) to be divided up among multiple Data DSAs.  This can have a significant impact on performance by allowing more DSAs to handle the LDAP request load.

Data Partitioning will take advantage of all available hardware, whether extra CPUs on a single host or CPUs across multiple hosts.

More replicas you have would allow for more parallel writes to the directory.

In the environments that additional servers are available or additional resources (CPUs or Cores) on a host are not being leveraged, then you can consider configuring CA Directory Data Partitioning.

CA SSO Policy Server specific configurations:

Policy server uses a Connection pool model (default size 10): A connection pool is a cache of backend repository (Database/LDAP) connections maintained so that the connections can be reused when future requests to the backend repository are required.  Connection pools are used to enhance the performance of executing commands on a backend repository.

If you have 50 threads that require session store connection handle (default 10) and the LDAP operations takes time to complete processing; this can cause normal thread queue to build up.

What to look for:  Using policy server profiler logs, observe if a request has a delay before Lock LDAP handle – this would mean we need to increase the number of session store connections

An example of what to look at in the profiler logs for this type of event:

Example: 2 seconds wait for thread ID (TID) 3003599728 before this worker thread was able to get an LDAP connection handle is shown below:

[13:23:38.671][26918][3003599728][Leave function CSm_Auth_Message::SetAuthContext][Sm_Auth_Message.cpp:4280][CSm_Auth_Message::SetAuthContext]

[13:23:40.671][26918][3003599728][Lock LDAP handle. slot=0 ld=0xb3071fe0][LdapStore.cpp:375][Lock_LdapHandle]

Resolution: How to increase connections to Session store?  Modify the following registry key (NOTE: configure the value close to half the max worker thread value so you have at least one connection handle for every 2 worker threads. This recommendation would vary based on the hardware used and other environmental considerations. For optimal performance, we recommend testing these in a test environment and modify as needed to adjust to a given environment.)

HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\LdapSessionServer=240093784
MaxConnections=                          0xa; REG_DWORD

SMDeleteSession (Available on a need basis and can be obtained by opening a support issue)

CA SSO Policy Servers configured to use CA Directory as a session store are responsible for creating sessions and then cleaning up the expired sessions.  The Policy Servers create sessions when users authenticate, then periodically, the Policy Server will search for sessions that have expired and remove them.

In high volume environments the CA SSO Policy Servers tend to spend many more cycles creating/updating sessions than removing them.  Over a period of time, this will cause the directory to fill with primarily expired sessions.   To counter act this issue, CA offers a specialized JAVA tool that can be configured as cronjob or schedule task, tool called the SMDeleteSession.  This process remains running, all the while checking for expired sessions, then removes them.  It’s only function is to remove expired sessions. (NOTE Policy server session maintenance code will be modified in future release making this tool not needed even for extremely high volume/traffic sites)

NOTE: SMDeleteSession is not applicable for policy servers at or above CA SSO 12.52 SP1 CR5 on the SP1 stream. For 12.52 SP2 this tool may still be needed.

 

ADDITIONAL INFORMATION:

DISCLAIMER:

CA IS DELIVERING THE FILES "AS IS" AND CA MAKES NO WARRANTIES WITH RESPECT TO THE FILES, EXPRESS, IMPLIED OR ARISING BY CUSTOM OR TRADE USAGE, AND SPECIFICALLY MAKES NO WARRANTY OF MERCHANTABILITY OR OF FITNESS FOR ANY PARTICULAR PURPOSE.  CA's EXPRESS WARRANTIES SHALL NOT BE ENLARGED, DIMINISHED OR AFFECTED BY, AND NO OBLIGATION OR LIABILITY SHALL ARISE OUT OF, CA's RENDERING OF TECHNICAL OR OTHER ADVICE OR SERVICE IN CONNECTION WITH THE FILES.

IN NO EVENT SHALL CA BE LIABLE FOR ANY LOSS OR DAMAGE OF ANY KIND, INCLUDING BUT NOT LIMITED TO, SPECIAL, INDIRECT, PUNITIVE INCIDENTAL OR CONSEQUENTIAL DAMAGES, ARISING OUT OF THE DELIVERY, USE, EVALUATION, SUPPORT OR OPERATION OF THE FILES PROVIDED EVEN IF CA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Environment

Release: ETRSBB99000-12.52-SiteMinder-B to B
Component: