Policy Server in hung state when In-memory Tracing enabled
Article ID: 232908
Updated On:
Products
Issue/Introduction
Policy servers were upgraded from version 12.7.2 to 12.8sp4 in production moths ago.
Everything seems to be fine, but occasionally find one of the servers goes into a hung state. The smpolicysrv process is still running but is not accepting any traffic.
smps.log
[7692/140338626897664][Fri Jan 21 2022 08:12:01][CServer.cpp:8056][INFO][sm-Server-02360] Server 'Stats' command received
[7692/140340508903232][Fri Jan 21 2022 09:00:13][CServer.cpp:6118][INFO][sm-Server-02100] Thread 140340508903232 received signal, stopping...
Between 08:12:01 and 09:00:13, the server was hung.
last request in smtracedefault.log
[01/21/2022][08:10:04][08:10:04.988][][][][][][7692][140339784439552][s6618/r478][OnAccessAccept][][][][][][][][IsAuthorized.cpp:787][CSm_Az_Message::IsAuthorized][agent][][][domain][rule][][][][][][][][|x8Ax8A@xCF
You will notice the non displayable characters (depending on the text editor) in the very end of the log.
Sometimes, log will be cut off/stopped at the exact same location without non displayable characters in the very end.
[12/29/2021][08:09:25][08:09:25.281][][][][][][56855][140381984282368][s411459/r352][AuthorizeEx][][][][][][][][Sm_Az_Message.cpp:828][CSm_Az_Message::FormatAttribute][agent][][][domain][rule][][][][][][][][
It could also exhibit as the following use case flow:
[99937/140083420186368][Wed Jan 06 2021 12:14:49][Sm_Az_Message.cpp:219][ERROR][sm-IsAuthorized-00220] Bad s10450/r1385 request detected: error 'Cannot fetch agent ��<script>alert(203)</script>'
[99937/140083445364480][Wed Jan 06 2021 12:14:49][Sm_Az_Message.cpp:219][ERROR][sm-IsAuthorized-00220] Bad s10450/r1386 request detected: error 'Cannot fetch agent ��<script>alert(203)</script>'
Environment
Release : 12.8
Component : SITEMINDER -POLICY SERVER
Cause
Policy server hung is due to In-memory Buffer tracing was turned on in the Policy server.
Policy serve is getting hanged sometimes with Buffer tracing enabled and the issue got fixed in later releases(12.8.05).
32383452 DE485101 Policy Server logs shutdown messages in smaccess.log instead of smtracedefault.log when buffer tracing is enabled.
Resolution
By default, in-memory tracing is disabled in the Policy Server Management Console.
The immediate resolution is to disable the In-memory Buffer tracing from SmConsole window.
In-memory Tracing Registry Keys:
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/troubleshooting/in-memory-tracing-to-troubleshoot-policy-server-failures/in-memory-tracing-registry-keys.html
HKEY_LOCAL_MACHINE\SOFTWARE\Netegrity\SiteMinder\CurrentVersion\LogConfig=552095227
BufferedTracing= 0x1; REG_DWORD
Change the value from 1 to 0.
Because this setting changes sm.registry file and is related to memory, service restart is recommended to ensure the change takes effect immediately.
Additional Information
DE526216
DE490361
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/release-notes/service-packs/Defects-Fixed-in-12-8-05.html
In-memory Tracing Registry Keys:
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/troubleshooting/in-memory-tracing-to-troubleshoot-policy-server-failures/in-memory-tracing-registry-keys.html
Feedback
