CA Single Sign On Secure Proxy Server (SiteMinder)CA Single Sign On Agents (SiteMinder)CA Single Sign On Federation (SiteMinder)CA Single Sign On SOA Security Manager (SiteMinder)SITEMINDER
Issue/Introduction
We're running a Policy Server and this one hanged and we had to restart it to recover the service.
We can see that the Agent connection increased until reaching the Max Connection limit on the Policy Server :
[381580/3973188464][Sat Nov 16 2019 15:30:43][CServer.cpp:3264][ERROR][sm-Server-07017] Connection request rejected. Connection limit of 16000 exceeded.
at 15:22, we see a problem on the nfs services :
Nov 16 15:22:16 myserver kernel: nfs: server mynasserver not responding, still trying
Nov 16 15:34:46 myserver kernel: nfs: server mynasserver OK
Nov 16 15:34:46 myserver kernel: nfs: server mynasserver OK
dmesg
nfs: server mynasserver not responding, still trying nfs: server mynasserver not responding, still trying nfs: server mynasserver OK nfs: server mynasserver OK
The pkgapp reports that all thread are in __kernel_vsyscall :
pkgapp.backtrace
Core was generated by `smpolicysrv'. #0 0x00480430 in __kernel_vsyscall () Missing separate debuginfos, use: debuginfo-install d0g-smjre-1.0.0.6-1.el6.ubs.x86_64 glibc-2.12-1.212.el6.i686 gmp-4.3.1-7.el6_2.2.i686 libgcc-4.4.7-3.el6.i686 libidn-1.18-2.el6.i686 libstdc++-4.4.7-3.el6.i686 sssd-client-1.13.3-60.el6.i686 (gdb) ------------BEGIN WHERE-------------- (gdb) #0 0x00480430 in __kernel_vsyscall () #1 0x00abce66 in nanosleep () from /lib/libpthread.so.0 #2 0x00203960 in SleepEx(unsigned long, int) () from /opt/CA/siteminder/lib/libsmcommonutil.so #3 0x00d0e1cd in LoggerMonitoringThread(void*) () from /opt/CA/siteminder/lib/libsmutilities.so #4 0x00205f29 in BtThreadBase(ThreadArgs*) () from /opt/CA/siteminder/lib/libsmcommonutil.so #5 0x00ab5bc9 in start_thread () from /lib/libpthread.so.0 #6 0x0371007e in clone () from /lib/libc.so.6 (gdb) ------------END WHERE---------------- (gdb) (gdb) ------------INFO THREADS------------- (gdb) 165 Thread 0xf77336e0 (LWP 381580) 0x00480430 in __kernel_vsyscall ()
Cause
There appears to be an issue with your User Store(s) or network. All Ping threads have timed out and all Bind threads are reporting LDAP Error 91. Threads sit at a busy state attempting to connect. New threads pick up agent requests and also encounter the same issue. Eventually all threads are busy, repeatedly sitting and eventually timing out. This occurs as more requests come into the queue and the queue builds until the web agents themselves start to timeout.
[381580/3635592048][Sat Nov 16 2019 15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver1.mydomain.com:9999
[381580/3099265904][Sat Nov 16 2019 15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver2.mydomain.com:9999
[381580/3646081904][Sat Nov 16 2019 15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver1.mydomain.com:9999
[381580/3088776048][Sat Nov 16 2019 15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver2.mydomain.com:9999
[381580/3646081904][Sat Nov 16 2019 15:19:45][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver3.mydomain.com:9999
[381580/3099265904][Sat Nov 16 2019 15:19:45][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver4.mydomain.com:9999
[381580/3135105904][Sat Nov 16 2019 15:19:45][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server myldapserver5.mydomain.com : 9999. Error 91-Can't connect to the LDAP server
[381580/3145595760][Sat Nov 16 2019 15:19:45][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server myldapserver6.mydomain.com : 9999. Error 91-Can't connect to the LDAP server
[381580/2816473968][Sat Nov 16 2019 15:19:48][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server myldapserver7.mydomain.com : 9999. Error 91-Can't connect to the LDAP server
[381580/3120245616][Sat Nov 16 2019 15:19:48][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server myldapserver8.mydomain.com : 9999. Error 91-Can't connect to the LDAP server
[381580/3109755760][Sat Nov 16 2019 15:19:48][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server myldapserver11.mydomain.com : 9999. Error 91-Can't connect to the LDAP server
[381580/3166169968][Sat Nov 16 2019 15:19:49][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server myldapserver12.mydomain.com : 9999. Error 91-Can't connect to the LDAP server
[381580/3197692784][Sat Nov 16 2019 15:19:53][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver13.mydomain.com:9999
[381580/3187202928][Sat Nov 16 2019 15:19:53][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver14.mydomain.com:9999
[381580/3646081904][Sat Nov 16 2019 15:19:55][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server myldapserver3.mydomain.com : 9999. Error 91-Can't connect to the LDAP server
[381580/3099265904][Sat Nov 16 2019 15:19:55][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server myldapserver13.mydomain.com : 9999. Error 91-Can't connect to the LDAP server
[381580/3135105904][Sat Nov 16 2019 15:19:55][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver6.mydomain.com:9999
[381580/3145595760][Sat Nov 16 2019 15:19:55][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver5.mydomain.com:9999
[381580/2816473968][Sat Nov 16 2019 15:19:58][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver12.mydomain.com:9999
[381580/3109755760][Sat Nov 16 2019 15:19:58][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver8.mydomain.com:9999
[381580/3120245616][Sat Nov 16 2019 15:19:58][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver9.mydomain.com:9999
[381580/3166169968][Sat Nov 16 2019 15:19:59][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at myldapserver7.mydomain.com:9999
How can we solve this ?
Environment
Policy Server 12.52SP1CR06 on RedHat 6;
Resolution
Investigate the network connectivity between the Policy Server and all LDAP servers at time of the issue;