R12.52 reaching max connection

book

Article ID: 185751

calendar_today

Updated On:

Products

CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On Agents (SiteMinder) CA Single Sign On Federation (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) SITEMINDER

Issue/Introduction


We're running a Policy Server and this one hanged and we had to
restart it to recover the service.

We can see that the Agent connection increased until reaching the Max
Connection limit on the Policy Server :

connections.txt

  21:15:03 Current=4648 Max=12981 Limit=16000 Exceeded limit=0

  15:20:02 Current=3053 Max=12981 Limit=16000 Exceeded limit=0
  15:25:02 Current=6275 Max=12981 Limit=16000 Exceeded limit=0
  15:30:02 Current=14677 Max=14677 Limit=16000 Exceeded limit=0
  15:35:02 Current=16000 Max=16000 Limit=16000 Exceeded limit=14443

And smps.log reported :

smps.log :

  [381580/3719510896][Sat Nov 16 2019
  15:30:02][CServer.cpp:4795][INFO][sm-Server-02040] Connections:
  Current=14677 Max=14677 Limit=16000 Exceeded limit=0

  [381580/3973188464][Sat Nov 16 2019
  15:30:43][CServer.cpp:3264][ERROR][sm-Server-07017] Connection
  request rejected. Connection limit of 16000 exceeded.

at 15:22, we see a problem on the nfs services :

  Nov 16 15:22:16 myserver kernel: nfs: server mynasserver not responding, still trying

  Nov 16 15:34:46 myserver kernel: nfs: server mynasserver OK

  Nov 16 15:34:46 myserver kernel: nfs: server mynasserver OK

dmesg

  nfs: server mynasserver not responding, still trying
  nfs: server mynasserver not responding, still trying
  nfs: server mynasserver OK
  nfs: server mynasserver OK

The pkgapp reports that all thread are in __kernel_vsyscall :

pkgapp.backtrace

  Core was generated by `smpolicysrv'.
  #0  0x00480430 in __kernel_vsyscall ()
  Missing separate debuginfos, use: debuginfo-install d0g-smjre-1.0.0.6-1.el6.ubs.x86_64 glibc-2.12-1.212.el6.i686 gmp-4.3.1-7.el6_2.2.i686 libgcc-4.4.7-3.el6.i686 libidn-1.18-2.el6.i686 libstdc++-4.4.7-3.el6.i686 sssd-client-1.13.3-60.el6.i686
  (gdb) ------------BEGIN WHERE--------------
  (gdb) #0  0x00480430 in __kernel_vsyscall ()
  #1  0x00abce66 in nanosleep () from /lib/libpthread.so.0
  #2  0x00203960 in SleepEx(unsigned long, int) () from /opt/CA/siteminder/lib/libsmcommonutil.so
  #3  0x00d0e1cd in LoggerMonitoringThread(void*) () from /opt/CA/siteminder/lib/libsmutilities.so
  #4  0x00205f29 in BtThreadBase(ThreadArgs*) () from /opt/CA/siteminder/lib/libsmcommonutil.so
  #5  0x00ab5bc9 in start_thread () from /lib/libpthread.so.0
  #6  0x0371007e in clone () from /lib/libc.so.6
  (gdb) ------------END WHERE----------------
  (gdb) 
  (gdb) ------------INFO THREADS-------------
  (gdb)   165 Thread 0xf77336e0 (LWP 381580)  0x00480430 in __kernel_vsyscall ()

Cause


There appears to be an issue with your User Store(s) or network. All
Ping threads have timed out and all Bind threads are reporting LDAP
Error 91.  Threads sit at a busy state attempting to connect.  New
threads pick up agent requests and also encounter the same issue.
Eventually all threads are busy, repeatedly sitting and eventually
timing out.  This occurs as more requests come into the queue and the
queue builds until the web agents themselves start to timeout.

  [381580/3635592048][Sat Nov 16 2019
  15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver1.mydomain.com:9999

  [381580/3099265904][Sat Nov 16 2019
  15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver2.mydomain.com:9999

  [381580/3646081904][Sat Nov 16 2019
  15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver1.mydomain.com:9999

  [381580/3088776048][Sat Nov 16 2019
  15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver2.mydomain.com:9999

  [381580/3646081904][Sat Nov 16 2019
  15:19:45][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver3.mydomain.com:9999

  [381580/3099265904][Sat Nov 16 2019
  15:19:45][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver4.mydomain.com:9999

  [381580/3135105904][Sat Nov 16 2019
  15:19:45][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370]
  SmDsLdapConnMgr Bind. Server
  myldapserver5.mydomain.com : 9999. Error 91-Can't connect to
  the LDAP server

  [381580/3145595760][Sat Nov 16 2019
  15:19:45][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370]
  SmDsLdapConnMgr Bind. Server
  myldapserver6.mydomain.com : 9999. Error 91-Can't connect to the
  LDAP server

  [381580/2816473968][Sat Nov 16 2019
  15:19:48][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370]
  SmDsLdapConnMgr Bind. Server
  myldapserver7.mydomain.com : 9999. Error 91-Can't connect to
  the LDAP server

  [381580/3120245616][Sat Nov 16 2019
  15:19:48][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370]
  SmDsLdapConnMgr Bind. Server
  myldapserver8.mydomain.com : 9999. Error 91-Can't connect to the
  LDAP server

  [381580/3109755760][Sat Nov 16 2019
  15:19:48][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370]
  SmDsLdapConnMgr Bind. Server
  myldapserver11.mydomain.com : 9999. Error 91-Can't connect to
  the LDAP server

  [381580/3166169968][Sat Nov 16 2019
  15:19:49][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370]
  SmDsLdapConnMgr Bind. Server
  myldapserver12.mydomain.com : 9999. Error 91-Can't connect to the
  LDAP server

  [381580/3197692784][Sat Nov 16 2019
  15:19:53][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver13.mydomain.com:9999

  [381580/3187202928][Sat Nov 16 2019
  15:19:53][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver14.mydomain.com:9999

  [381580/3646081904][Sat Nov 16 2019
  15:19:55][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370]
  SmDsLdapConnMgr Bind. Server
  myldapserver3.mydomain.com : 9999. Error 91-Can't connect to
  the LDAP server

  [381580/3099265904][Sat Nov 16 2019
  15:19:55][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370]
  SmDsLdapConnMgr Bind. Server
  myldapserver13.mydomain.com : 9999. Error 91-Can't connect to the
  LDAP server

  [381580/3135105904][Sat Nov 16 2019
  15:19:55][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver6.mydomain.com:9999

  [381580/3145595760][Sat Nov 16 2019
  15:19:55][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver5.mydomain.com:9999

  [381580/2816473968][Sat Nov 16 2019
  15:19:58][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver12.mydomain.com:9999

  [381580/3109755760][Sat Nov 16 2019
  15:19:58][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver8.mydomain.com:9999

  [381580/3120245616][Sat Nov 16 2019
  15:19:58][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver9.mydomain.com:9999

  [381580/3166169968][Sat Nov 16 2019
  15:19:59][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280]
  SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at
  myldapserver7.mydomain.com:9999

How can we solve this ?

Environment


Policy Server 12.52SP1CR06 on RedHat 6;

Resolution


Investigate the network connectivity between the Policy Server and all
LDAP servers at time of the issue;
Powered by Wolken Software