Buffered Tracing may cause a Siteminder Policy Server Crash in Rare Circumstances

Products

• SITEMINDER

Issue/Introduction

 

When writing a trace log using the Buffered Tracing feature and the trace configuration ('smtracedefault.txt') is modified, the Siteminder Policy Server may experience a race condition causing the 'smpolicysrv' process to stop abruptly.

[pstack]

--------------------------------------------------------------------------------------------------

/opt/CA/siteminder/lib/libsmerrlog.so(+0x1844e)[0x7fa21f35244e]
/opt/java/jre/lib/amd64/server/libjvm.so(+0x922b1b)[0x7fa147121b1b]
/opt/java/jre/lib/amd64/server/libjvm.so(JVM_handle_linux_signal+0x27d)[0x7fa14712682d]
/opt/java/jre/lib/amd64/server/libjvm.so(+0x91a3a8)[0x7fa1471193a8]
/lib64/libpthread.so.0(+0xf630)[0x7fa21da96630]
/lib64/libc.so.6(+0x140dd0)[0x7fa21b221dd0]
/opt/CA/siteminder/lib/libsmerrlog.so(_ZN13CSmFileOutput8RolloverEv+0xa2)[0x7fa21f352df2]
/opt/CA/siteminder/lib/libsmerrlog.so(_ZN13CSmFileOutput32RollOverCheck_LogMessageBufferedEPKcm+0xbf)[0x7fa21f353eff]
/opt/CA/siteminder/lib/libsmerrlog.so(_Z19SmTraceWriterThreadPv+0xf3)[0x7fa21f354013]
/opt/CA/siteminder/lib/libGCL.so(_ZN9CpaThread15ThreadFuncWedgeEPv+0x8e)[0x7fa2233b390e]
Received Signal: 11

--------------------------------------------------------------------------------------------------

Environment

 

Product: Siteminder

Component: Policy Server

Versions Impacted: r12.8 - r12.8.7

Cause

 

This behavior can occur under the following combination of variables:

• Policy Server is running
• Policy Server Profiler Tracing is enabled
• Buffer Tracing is enabled
• The Policy Server Profiler configuration file ('smtracedefault.txt') is modified

The Policy Server can encounter a race condition causing the 'smpolicysrv' process to abruptly stop.

Resolution

 

This issue is resolved in Siteminder Policy Server r12.8.8. 

 

###### WORKAROUND #######

If 'Buffered Tracing' is enabled on the Policy Server, you will need to stop the policy server prior to making changes to the Policy Server profiler settings.  This is regardless of whether you are using the AdminUI or 'smtracedefault.txt' file. Then start the policy server once the changes are made.

Alternatively, you could disable buffered tracing and leave profiler tracing (policy server) running.  This would require a one-time restart of the policy server, however it would allow you to make changes to the policy server tracing without crashing the policy server.

 

Disable Buffered Tracing Using the Siteminder SMConsole

1) Logon to the Siteminder Policy Server

2) Launch the SMConsole

3) Go to the Profiler tab

4) remove the flag for "Buffered Tracing"

NOTE: This value can only be toggled if 'Enable Profiling' is also enabled.

WARNING:  IF 'Buffered Tracing' is enabled and 'Enable Profiling' is disabled DO NOT set the flag for 'Enable Profiling' in order to remove the 'Buffered Tracing' flag.  Instead, disable 'Buffered Tracing' through the Siteminder Registry.  (See registry instructions (below)).

 

Disable Buffered Tracing Using the Siteminder Registry

[LINUX]

1) Logon to the Siteminder Policy Server

2) Open the Siteminder registry file with a text editor

<Install_Dir>/siteminder/registry/sm.registry

3) Browse to the following registry hive:

HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\LogConfig

4) Set the following:

BufferedTracing = 0x0

5) Save changes to the 'sm.registry' file

6) Stop and start the Siteminder Policy Server.

7) Repeat on all Policy Servers

 

[WINDOWS]

1) Logon to the Siteminder Policy Server

2) Launch the Windows Registry Editor

Start -> Run: 'regedit.exe'

3) Browse to the following registry hive:

HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\LogConfig

4) Set the following:

BufferedTracing = 0x0

5) Stop and start the Siteminder Policy Server.

7) Repeat on all Policy Servers