Secure Proxy Server hangs under load

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• CA Single Sign-On

Issue/Introduction

ISSUE:

Secure Proxy Server becomes unresponsive as load increased to a certain level and it needs to be restarted to return to operation state.

Following errors are logged corresponding to the event:

<Please see attached file for image>

CAUSE:

Following are some settings on SPS: 

 

 

<Please see attached file for image>

Customer is using ‘spsuser’ account to run both Apache and Tomcat processes. ‘spsuser’ account has max user processes (ulimit –u) set to 1024. Hence, the user is restricted to a total of 1024 threads/processes across the whole Linux server.

As load increases, Apache spawns more child processes and each child process takes up a further 27 threads. However, the Tomcat process takes up to a total of ~410 threads. That only leaves 614 threads/processes - which is equivalent to 22 httpd processes (providing the user is not running any other process or command). When 22 httpd processes are reached and Tomcat process logged the “unable to create new native thread” exception and SPS hangs.

<Please see attached file for image>

The most important directives used to control Apache MPM worker are ThreadsPerChild, which controls the number of threads deployed by each child process and MaxRequestWorkers, which controls the maximum total number of threads that may be launched. The maximum number of clients that may be served simultaneously (i.e., the maximum total number of threads in all processes) is determined by the MaxRequestWorkers directive. The maximum number of active child processes is determined by the MaxRequestWorkers directive divided by the ThreadsPerChild directive. 

With MaxRequestWorkers=400 and ThreadsPerChild=25 in httpd-mpm.conf, we expect maximum of 16 active child processes. 

However, we observed more than 16 httpd processes during load test. This is due to the conflicting directives in httpd-mpm.conf -- MaxClients vs MaxRequestWorkers. MaxRequestWorkers was called MaxClients before version 2.3.13. However, the old name is still supported. 

 

RESOLUTION:

Options to overcome the issue:

• increase the max user processes (ulimit -u)

 

• remove the conflicting directive (MaxClients) from httpd-mpm.conf to restrict the number of active Apache child processes

 

Environment

Release: ETRSBB99000-12.52-SiteMinder-B to B
Component: