search cancel

Error: No connections available for Found In-Use = true in SPS

book

Article ID: 240160

calendar_today

Updated On:

Products

SITEMINDER CA Single Sign On Secure Proxy Server (SiteMinder)

Issue/Introduction

 

When running CA Access Gateway (SPS), this one reports an error:

     No connections available for [{s}->https://mysps.mydomain.com:443] Found In-Use = true]

This error occurs with the integration of this specific application.

 

Cause

 

The SPS get started at 04:51:54:

sps-webagent.log:

  [1860/4304][Sun Apr 03 2022 04:51:54] CA Access Gateway Agent, Version 12.8 QMR04, Update None, Label 2278

sps-webagent-trace.log:


  
  [04/03/2022][04:51:54][1860][4304][][CSmLowLevelAgent.cpp:6279][InitializeTracingMT]
  [][][][][][][][][][][][][][][][Tracing initialized.]

and the first error occurrence appears at 05:12:32 :

  [04/03/2022][05:12:32][1860][8812][16f9d0a3-ede8aa60-efdf14f1-d03e9208-fc993223-77d7]
  [com.ca.proxy.connectionpool.impl.AuthNConnectionPoolImpl][getConnection][][][][][]
  [][][][][][][][][][]
  [No connections available for [{s}->https://mysps.mydomain.com:443] Found In-Use = true]
Before that first occurrence, the SPS already has intermittent problem
to get connection with the backend server, as with
myothersps.mydomain.com for instance, which failed after 3 tentatives:
  
  [04/03/2022][04:58:37][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [ProxyValve.java][ProxyValve::invoke][][][][][][][][][][][][][][][][Entering the agent.]

  [04/03/2022][04:58:37][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [CSmHttpPlugin.cpp:869][CSmHttpPlugin::ProcessResource][][][][agent][][][][][][][][]
  [][][][Resolved URL: '/myapp/myapp.html'.]
  
  [04/03/2022][04:58:37][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [CSmHttpPlugin.cpp:1001][CSmHttpPlugin::ProcessResource][][[email protected] ][][agent]
  [/myapp/myapp.html][][][][][][][][][][][Resolved METHOD: 'GET'.]
  
  [04/03/2022][04:58:37][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [SmAgentAPI.cpp:1761][Sm_AgentApi_IsProtected][][https://myothersps.mydomain.com][][agent]
  [/myapp/myapp.html][][][][][][][][][][][]
  
  [04/03/2022][04:58:37][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [CSmLowLevelAgent.cpp:577][IsResourceProtected][][*][][agent]
  [/myapp/myapp.html][][][][][][][][][][][Resource is not protected from Policy Server.]
  
  [04/03/2022][04:58:37][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [SmProxyRules.java][SmProxyRules.processRules][][][][][][][][][][][][][][][]
  [Dispatching to service FORWARD with url https://10.0.0.1/myapp/myapp.html]
  
  [04/03/2022][04:58:37][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [com.ca.proxy.apache.httpclient.SPSClient][execute][][][][][][][][][][][][][][][]
  [Sending request to backend = 10.0.0.1 url = https://10.0.0.1/myapp/myapp.html]
  
  [04/03/2022][04:58:37][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [com.ca.proxy.apache.httpclient.conn.factory.SPSConnectionFactory][openConnection()]
  [][][][][][][][][][][][][][][][Connecting to /10.0.0.1:443]

  [04/03/2022][04:58:47][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [com.ca.proxy.apache.httpclient.SPSClient][execute][][][][][][][][][][][][][][][]
  [Retrying to send the request to backend web server.Retry count: 1]

  [04/03/2022][04:58:47][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [com.ca.proxy.apache.httpclient.SPSClient][execute][][][][][][][][][][][][][][][]
  [Sending request to backend = 10.0.0.1 url = https://10.0.0.1/myapp/myapp.html]

  [04/03/2022][04:58:47][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [com.ca.proxy.apache.httpclient.conn.SPSConnectionManager][requestConnection(): ][]
  [][][][][][][][][][][][][][][Get connection: {s}->https://10.0.0.1:443, timeout = 180000]

  [04/03/2022][04:58:57][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [com.ca.proxy.apache.httpclient.SPSClient][execute][][][][][][][][][][][][][][][]
  [Retrying to send the request to backend web server.Retry count: 2]

  [04/03/2022][04:58:57][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [com.ca.proxy.apache.httpclient.SPSClient][execute][][][][][][][][][][][][][][][]
  [Sending request to backend = 10.0.0.1 url = https://10.0.0.1/myapp/myapp.html]

  [04/03/2022][04:58:57][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [com.ca.proxy.apache.httpclient.conn.SPSConnectionManager][requestConnection(): ][]
  [][][][][][][][][][][][][][][Get connection: {s}->https://10.0.0.1:443, timeout = 180000]

  [04/03/2022][04:58:57][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [com.ca.proxy.apache.httpclient.conn.factory.SPSConnectionFactory][openConnection()]
  [][][][][][][][][][][][][][][][Connecting to /10.0.0.1:443]

  [04/03/2022][04:59:07][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [com.ca.proxy.apache.httpclient.SPSClient][execute][][][][][][][][][][][][][][][]
  [Retrying to send the request to backend web server.Retry count: 3]
  
  [04/03/2022][04:59:07][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [com.ca.proxy.apache.httpclient.SPSClient][execute][][][][][][][][][][][][][][][]
  [Tried to send the request to backend web server three times.Throwing the exception to client. ]

  [04/03/2022][04:59:07][1860][3004][78cf9c61-d361cfae-9a61b3c9-8c0c7484-25852099-eff]
  [Noodle.java][Noodle::doGet][][][][][][][][][][][][][][][]
  [org.apache.http.client.ClientProtocolException
  at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:839)]

  [04/03/2022][04:59:08][1860][3004][18155d83-b7d04b90-9c585ec4-17959c27-f35ea6e7-ad40]
  [ProxyValve.java][ProxyValve::invoke][][][][][][][][][][][][][][][][Entering the agent.]

So going on that way, SPS needs to consistently allocate new
connections until it reaches the limit and we see the error:

  No connections available for

This backend server doesn't have always connectivity problems. At
04:58:37, the myothersps.mydomain.com / 10.0.0.1 replies within the
same second.

  [04/03/2022][04:58:37][1860][8952][1a03da5d-3a3761c5-799d6ba0-7bf317fd-fa6ca99d-8dd]
  [SmProxyRules.java][SmProxyRules.processRules][][][][][][][][][][][][][][][]
  [Proxy Rules Debug: Evaluating Condition: Type: null, Case: 'myothersps.mydomain.com:443'
  equals 'myothersps.mydomain.com:443']

  [04/03/2022][04:58:37][1860][8952][1a03da5d-3a3761c5-799d6ba0-7bf317fd-fa6ca99d-8dd]
  [SmProxyRules.java][SmProxyRules.processRules][][][][][][][][][][][][][][][]
  [Dispatching to service FORWARD with url https://10.0.0.1/myotherapp/myotherapp.html]
  
  [04/03/2022][04:58:37][1860][8952][1a03da5d-3a3761c5-799d6ba0-7bf317fd-fa6ca99d-8dd]
  [com.ca.proxy.apache.httpclient.SPSClient][execute][][][][][][][][][][][][][][][]
  [Sending request to backend = 10.0.0.1 url = https://10.0.0.1/myotherapp/myotherapp.html]
  
  [04/03/2022][04:58:37][1860][8952][1a03da5d-3a3761c5-799d6ba0-7bf317fd-fa6ca99d-8dd]
  [com.ca.proxy.apache.httpclient.conn.SPSConnectionManager][requestConnection(): ][]
  [][][][][][][][][][][][][][][Get connection: {s}->https://10.0.0.1:443, timeout = 180000]

  [04/03/2022][04:58:37][1860][8952][1a03da5d-3a3761c5-799d6ba0-7bf317fd-fa6ca99d-8dd]
  [com.ca.proxy.apache.httpclient.conn.factory.SPSConnectionFactory][openConnection()]
  [][][][][][][][][][][][][][][][Connecting to /10.0.0.1:443]

  [04/03/2022][04:58:37][1860][8952][1a03da5d-3a3761c5-799d6ba0-7bf317fd-fa6ca99d-8dd]
  [com.ca.proxy.apache.httpclient.SPSClient][execute][][][][][][][][][][][][][][][]
  [Response status code from backend webserver is 200]

 

Environment

 

CA Access Gateway (SPS) 12.8SP4 on RedHat 7

 

Resolution

 

The connectivity problem is intermittent, and not only limited to
mysps.mydomain.com.

Even before the first error occurrence, a bunch of connections
tentative fail, as the SPS tries many times to reach the backend
server without success. But not always! Some connections are done
quickly, but not all!

When restarting SPS, the entire connection pool is available, but it
becomes busy within 20 minutes because it accumulates the calls which
take a long time to get a connection with the backend servers.

To fix this issue, you have to investigate the network and the backend
servers' responsiveness and connectivity.