SiteMinder : Agent API Failure with ODBC Deadlock on R12.8.x of Policy server

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On Agents (SiteMinder)
• CA Single Sign On Federation (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• SITEMINDER

Issue/Introduction

===Issue Symptom===

Customer was observing the Agent API failure on the SSO agent side and Access Gateway side. 

[22379/2695752560][Thu Sep 10 2020 05:01:41][agentcommon][INFO][sm-FedClient-00010] The doManagement Thread failed as the Policy Server could not be reached. Reason: Agent API Failure
[22379/2695752560][Thu Sep 10 2020 05:11:41][agentcommon][INFO][sm-FedClient-00010] The doManagement Thread failed as the Policy Server could not be reached. Reason: Agent API Failure

====Observations====

Usually the Agent API fails because of the  network communication issues b/w agent to policy servers communication, like latency and glitches on the network layer. 

There was no policy server connections and thread were stable for couple of days post issue was reported and abruptly few policy servers were reaching max configured connections and max thread counts. 

 

Environment

Release : 12.8.x

Component : SITEMINDER POLICY SERVER

Cause

====Investigation====

All the Webagents /SPS servers configured to connect to policy server listed in SmHost.conf file were not able to talk to policy servers because of the there were no policy server connections / threads available for new incoming request and cause in Agent API failures and SMSESSION management failures. 

====Reason====

Customer uses ODBC user directory and all the ODBC connections with policy servers was going to still state or responding very slow to new incoming requests and accumulating the number of incoming agent requests causing policy server deadlock situation and all the existing connections / sessions with policy servers are broken on refresh interval.

Resolution

-> Check if the Policy server connections reaching maximum by executing below command on the Policy server.

smpolicysrv -stats

Note : Policy server connection stats are recorded in smps.log file and review the smps.log file post executing the "smpolicysrv -stats" command.

-> Check the network latency / routing table to see if the Access Gateway has problems reaching Policy server and involve the network team accordingly to fix the network issues.