Policyserver Auto restart

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On Agents (SiteMinder)
• CA Single Sign On Federation (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• SITEMINDER

Issue/Introduction

We're running a Policy Server and this one reaches the max thread
limit and crashes. Before the crash, we've seen these statistics in
the logs :

           <THREAD_MSG_QUEUE>
                <MSG_TOTALS>    40833464 </MSG_TOTALS>
                <THROUGHPUT>    1.85495 </THROUGHPUT>
                <AVG_RESPONSE_TIME>    19.1252 </AVG_RESPONSE_TIME>
                <AVG_WAIT_TIME_IN_QUEUE>    0.0132999 </AVG_WAIT_TIME_IN_QUEUE>
                <MAX_HIGH_DEPTH>     4 </MAX_HIGH_DEPTH>
                <MAX_NORM_DEPTH>     60 </MAX_NORM_DEPTH>
                <MAX_MSG_DEPTH>     60 </MAX_MSG_DEPTH>
                <CURRENT_HIGH_DEPTH>     0 </CURRENT_HIGH_DEPTH>
                <CURRENT_NORM_DEPTH>     0 </CURRENT_NORM_DEPTH>
                <CURRENT_MSG_DEPTH>     0 </CURRENT_MSG_DEPTH>
            </THREAD_MSG_QUEUE>
            <THREADS_LIMIT> 150 </THREADS_LIMIT>
            <THREADS_MAX>   150 </THREADS_MAX>
            <THREADS_CURRENT> 150 </THREADS_CURRENT>
            <THREADS_BUSY> 0 </THREADS_BUSY>

We'd like to know :

 1 - What means each of the parameters;
 2 - What means AVG_WAIT_TIME_IN_QUEUE;
 3 - What are the factors causing the thread count increase;

Environment

Policy Server all versions

Resolution

At first glance, here are the answers :

1. 

Overall, a message is an Agent Request. There are 2 types :

  - High requests (Agent connection)
  - Normal requests (isProtected, isAuthenticated and isAuthorized)

Usually, a new thread will be created if there's none available when
Agent sends a connection request.

So said : 

  MSG_TOTALS (40833464)
 
  # Amount of times a message was removed from the queue since the last
  # Policy Server start;

  THROUGHPUT (1.85495 message by ms)

  # MSG_TOTALS divided by the time elapsed since the last start of the
  # Policy Server;

  AVG_RESPONSE_TIME (19.1252 ms by message)

  # Cumulative Response Time (time taken by a thread to process the
  # message completely) divided by MSG_TOTALS (ms);

  AVG_WAIT_TIME_IN_QUEUE (0.0132999 ms by message)
 
  # Cumulative Waiting time of the messages in queue divided by
  # MSG_TOTALS (ms);

  MAX_HIGH_DEPTH (4)
  
  # Maximum amount of High Priority messages in the queue;

  MAX_NORM_DEPTH (60)

  # Maximum amount of Normal Priority messages in the queue;

  MAX_MSG_DEPTH (60)

  # Maximum number of messages on the queue (High or Normal Priority
  # queues)

  CURRENT_HIGH_DEPTH (0)

  # Amount of High Priority Queue (Agent Connections) messages waiting
  # to be handled by a Thread at the current time;

  CURRENT_NORM_DEPTH (0)

  # Amount of Normal Priority Queue messages waiting to be handled by
  # a Thread at the current time;

  CURRENT_MSG_DEPTH (0)

  # CURRENT_HIGH_DEPTH and CURRENT_NORM_DEPTH combined at current time;

  THREADS_LIMIT (150)

  # The Maximum Threads you have defined in the smconsole, in
  # "Settings" tab;

  THREADS_MAX (150)

  # The highest amount of Threads initialized since last Policy Server
  # start;

  THREADS_CURRENT (150)

  # The amount of Threads running;

  THREADS_BUSY (150)

  # The amount of threads actively processing a request;

2. 

See answer in 1.

3.

Usually, a new thread will be created if there's none available when
Agent sends a connection request.

You mentioned :

  "recently we are observing the Policy server auto restart"

If the Policy Server is 12.52SP1, then it is a 32bit process, for
which you have defined 150 threads. This is much too high amount of
threads and there are big chances that the Policy Server restarted
because of lack of memory.

Ususally, you should keep the out of the box value for max thread and
add new Policy Servers if this is not enough.

Each thread starts with more than 10 mb of memory, and grow further
when processing. To that you should add all the memory the java
processing take.