Policy Server abnormal use of the CPU and hanging requests

Products

• SITEMINDER

Issue/Introduction

 

When running a Policy Server, it may be observed that the Policy Server stopped responding to the Web Agents when one of the Policy Server processes was taking time to process 1 request, using a lot of CPU cycles. So all the other requests were waiting for this process to end.    

Usually, a restart of the Policy Server helps to recover the service.

 

Resolution

 

At first glance, it's possible that the Policy Server used abnormal CPU cycle amounts when regular expressions are in use in Realm and an unexpected URL is requested. In some cases, a Custom Authentication Scheme or a Custom Active Expression code might fall into an infinite loop, and high CPU usage from the Policy Server process could be observed too.

Also, can Policy Server use a lot of CPU cycles by problems in cleaning the server commands (1).

Policy Server can be seen using a lot of CPU when getting data from ODBC Store (2).

Overall, to understand what happens and what might cause higher CPU usage, it's recommended to enable the full Policy Server traces (Profiler) (3)(4).

And then once having collected those traces, run the Policy Server traces analyzer against those traces to isolate the thread that was still processing while the other threads were waiting.

 

Additional Information

 

(1)

    Policy Server : CPU Spike : CleanServerCmds Error

      When running a Policy Server it can be observed that the Policy Server
      is consuming high CPU. A restart of the Policy Server might be needed
      in order to solve the CPU spike.

(2)

    Policy server hangs and crashes with very high memory and CPU utilization.

      If an underlying ODBC data store returns huge amount of data (e.g.-
      configured response attribute returns huge data from underlying odbc
      user store) then Policy server may hang/crash while trying to fetch
      the larger number of records from underlying ODBC database.

      This would result in very high CPU utilization and memory utilization.

(3)

    HOW TO ENABLE POLICY SERVER TRACING LOG & ADD ALL THE COMPONENTS/DATA FIELDS
    

(4)

  
    Configure the Policy Server Profiler
    

(5)

    Siteminder Policy Trace Analysis