Policy Server reaches max connection
search cancel

Policy Server reaches max connection

book

Article ID: 185751

calendar_today

Updated On:

Products

CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On Agents (SiteMinder) CA Single Sign On Federation (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) SITEMINDER

Issue/Introduction


When running a Policy Server and this one hanged, a restart of the Policy Server has been needed to recover the service.

The Agent connections increased until reaching the Max Connection limit on the Policy Server:

  21:15:03 Current=4648 Max=12981 Limit=16000 Exceeded limit=0
  15:20:02 Current=3053 Max=12981 Limit=16000 Exceeded limit=0
  15:25:02 Current=6275 Max=12981 Limit=16000 Exceeded limit=0
  15:30:02 Current=14677 Max=14677 Limit=16000 Exceeded limit=0
  15:35:02 Current=16000 Max=16000 Limit=16000 Exceeded limit=14443

And the smps.log reported:

smps.log :

  [381580/3719510896][Sat Nov 16 2019 15:30:02][CServer.cpp:4795][INFO][sm-Server-02040] Connections: Current=14677 Max=14677 Limit=16000 Exceeded limit=0
  [381580/3973188464][Sat Nov 16 2019 15:30:43][CServer.cpp:3264][ERROR][sm-Server-07017] Connection request rejected. Connection limit of 16000 exceeded.

 

Environment

 

Policy Server on RedHat 6;

 

Cause

 

There can be many factors that impact processing time for a transaction by a thread which in this particular case was caused by backend user Directory where the Ping thread was reporting LDAP error 91 (Failed to connect to server).

Threads sit in a busy state attempting to connect.

New threads pick up agent requests and also encounter the same issue.

Eventually, all threads are busy, repeatedly sitting and eventually timing out.  This occurs as more requests come into the queue and the queue builds until the web agents themselves start to timeout.

  [381580/3635592048][Sat Nov 16 2019 15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap1.example.com:9999
  [381580/3099265904][Sat Nov 16 2019 15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap2.example.com:9999
  [381580/3646081904][Sat Nov 16 2019 15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap1.example.com:9999
  [381580/3088776048][Sat Nov 16 2019 15:19:25][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap2.example.com:9999
  [381580/3646081904][Sat Nov 16 2019 15:19:45][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap3.example.com:9999
  [381580/3099265904][Sat Nov 16 2019 15:19:45][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap4.example.com:9999
  [381580/3135105904][Sat Nov 16 2019 15:19:45][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server ldap5.example.com : 9999. Error 91-Can't connect to the LDAP server
  [381580/3145595760][Sat Nov 16 2019 15:19:45][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server ldap6.example.com : 9999. Error 91-Can't connect to the LDAP server
  [381580/2816473968][Sat Nov 16 2019 15:19:48][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server ldap7.example.com : 9999. Error 91-Can't connect to the LDAP server
  [381580/3120245616][Sat Nov 16 2019 15:19:48][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server ldap8.example.com : 9999. Error 91-Can't connect to the LDAP server
  [381580/3109755760][Sat Nov 16 2019 15:19:48][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server ldap11.example.com : 9999. Error 91-Can't connect to the LDAP server
  [381580/3166169968][Sat Nov 16 2019 15:19:49][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server ldap12.example.com : 9999. Error 91-Can't connect to the LDAP server
  [381580/3197692784][Sat Nov 16 2019 15:19:53][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap13.example.com:9999
  [381580/3187202928][Sat Nov 16 2019 15:19:53][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap14.example.com:9999
  [381580/3646081904][Sat Nov 16 2019 15:19:55][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server ldap3.example.com : 9999. Error 91-Can't connect to the LDAP server
  [381580/3099265904][Sat Nov 16 2019 15:19:55][SmDsLdapConnMgr.cpp:909][ERROR][sm-Ldap-01370] SmDsLdapConnMgr Bind. Server ldap13.example.com : 9999. Error 91-Can't connect to the LDAP server
  [381580/3135105904][Sat Nov 16 2019 15:19:55][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap6.example.com:9999
  [381580/3145595760][Sat Nov 16 2019 15:19:55][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap5.example.com:9999
  [381580/2816473968][Sat Nov 16 2019 15:19:58][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap12.example.com:9999
  [381580/3109755760][Sat Nov 16 2019 15:19:58][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap8.example.com:9999
  [381580/3120245616][Sat Nov 16 2019 15:19:58][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap9.example.com:9999
  [381580/3166169968][Sat Nov 16 2019 15:19:59][SmDsLdapConnMgr.cpp:647][ERROR][sm-Ldap-01280] SmDsLdapConnMgr (ldap_search_ext_s) in PingServer : Timed out at ldap7.example.com:9999

 

Resolution


Investigate the network connectivity between the Policy Server and all LDAP servers at the time of the issue;

 

Powered by Wolken Software