Siteminder Policy Server machines show high CPU every night
Article ID: 220128
Updated On:
Products
Issue/Introduction
How to identify High CPU consumption on Servers where Siteminder Policy Servers are installed?
Cause
When it comes to CPU consumption on the servers hosting Siteminder components (in this case Policy Server), it's essential to identify the process consuming the CPU.
listed below are a couple of command lines commands for Linux systems to help narrow this down.
1) Use the following command to identify what process is consuming the CPU
# top -d 5 -n 240 -b >> top.txt
The samples below from the above captured output shows that Siteminder process is not the one causing the High CPU in this case.
top - 00:45:48 up 2 days, 12:02, 1 user, load average: 4.38, 3.74, 2.30
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1105 root 20 0 2385168 1.3g 75592 S 76.5 16.8 53:42.80 sisamddaemon
3027 smuser 20 0 3661396 478784 47692 S 5.9 6.0 28:55.84 smpolicysrv
top - 00:45:53 up 2 days, 12:02, 1 user, load average: 4.43, 3.76, 2.31
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1105 root 20 0 2385200 1.3g 75592 S 80.2 16.8 53:46.82 sisamddaemon
3027 smuser 20 0 3661396 478784 47692 S 0.6 6.0 28:55.87 smpolicysrv
top - 00:45:58 up 2 days, 12:02, 1 user, load average: 4.64, 3.82, 2.34
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1105 root 20 0 2385296 1.3g 75592 S 80.1 16.8 53:50.84 sisamddaemon
3027 smuser 20 0 3661396 478784 47692 S 0.4 6.0 28:55.89 smpolicysrv
2) In the case where the Siteminder smpolicysrv process is the one that is utilizing a High CPU, use the below to debug.
ps -Lp $(pgrep -x smpolicysrv) -o comm,pid,lwp,pcpu
The above will give all the running threads within the smpolicysrv process and which thread is consuming the CPU.
Resolution
In this situation, the first thing to do is to check which program is consuming the CPU: Policy Server or another program.
In the sample above, it's to be noticed that Policy Server uses 0.2 to 6% of the CPU while the program sisamddaemon uses between 76 and 80% of the CPU.
In this specific use case, investigate the reason why the sisamddaemon program uses such an amount of CPU. Consult the vendor if needed.
Enable full trace and capture it at the same time and isolate the thread in the Trace and see what activity this thread is performing (is it processing a custom code? is it deleting session from the Session Store ...).
In addition to the above, take a couple of pstack or dump the process using gcore and open a case with Support and upload the collected logs and output from the command line so they can assist in getting to the root cause of what activity the thread is performing.
Feedback
