We have noticed that our Policy Server is restarting from time to time after a crash, and reviewing the logs it seems the problem occurs when we have done changes in the Policy Store (like adding a new TrustedHost, etc). This does not happen every time we change the Policy Store, only sometimes.
We have found a coredump from the crash, and we see the following stack :
Core was generated by `smpolicysrv'.
Program terminated with signal 11, Segmentation fault.
#0 0x01e6c88e in CObjectCacheManager::RebuildIndex(CSmartPtr<CObjectCache>&) ()
(gdb) #0 0x01e6c88e in CObjectCacheManager::RebuildIndex(CSmartPtr<CObjectCache>&) ()
#1 0x01f284c0 in CSmObjStore::FindObjectByProp(CSmObjId::SmObjId_Class, unsigned int, CString const&, CSmObjBase const*&) () from /opt/siteminder/siteminder/lib/libsmobj.so
#2 0x01f28688 in CSmObjStore::FindAgentByName(CString const&, CSmObjDevice*&) ()
#3 0x0816fb14 in CSmAgentIdentity::FetchDevice() ()
#4 0x0816ed51 in CSmAgentIdentity::SetAgentIdentity(CString&, int, CString&) ()
#5 0x080b2b2a in CSm_Az_Message::ProcessMessage() ()
#6 0x081470e1 in CSmPolicyServer::vOnRequest(CClientSession const*, CString const&, unsigned int, CSmAgentTliPacket&, CSmAgentTliPacket&, int) ()
#7 0x0095b01c in CServer::ProcessRequest(CClientSession*, CString const&, unsigned int, CSmAgentTliPacket&, CSmAgentTliPacket&, int) () from /opt/siteminder/siteminder/lib/libsmutilities.so
#8 0x0093539c in CAgentMessageHandler::DoWork(unsigned char*, unsigned char*, int) ()
#9 0x0092c3ee in ThreadPool::Run(bool) () from /opt/siteminder/siteminder/lib/libsmutilities.so
#10 0x009dd096 in ThreadPoolBase::ThreadProc(void*) () from /opt/siteminder/siteminder/lib/libsmutilities.so
How can we fix this ?
This is caused by a race condition which provokes the crash when the secondary cache is being rebuilt, and when this occurs sometimes the Policy Server will crash because of a memory corruption of the secondary cache.
This issue is fixed in R12.52 SP1 CR02 to avoid that race condition to occur, so you need to upgrade your Policy Server to R12.52 SP1 CR02 to solve the issue.
The policy server crashes or hangs. Crashed policy servers restart, hung policy servers must be restarted.
This issue has been fixed.