Customer was observing the Agent API failure on the SSO agent side and Access Gateway side.
[22379/2695752560][Thu Sep 10 2020 05:01:41][agentcommon][INFO][sm-FedClient-00010] The doManagement Thread failed as the Policy Server could not be reached. Reason: Agent API Failure
[22379/2695752560][Thu Sep 10 2020 05:11:41][agentcommon][INFO][sm-FedClient-00010] The doManagement Thread failed as the Policy Server could not be reached. Reason: Agent API Failure
Usually the Agent API fails because of the network communication issues b/w agent to policy servers communication, like latency and glitches on the network layer.
There was no policy server connections and thread were stable for couple of days post issue was reported and abruptly few policy servers were reaching max configured connections and max thread counts.
Release : 12.8 SP3
Component : SITEMINDER POLICY SERVER
All the Webagents /SPS servers configured to connect to policy server listed in SmHost.conf file were not able to talk to policy servers because of the there were no policy server connections / threads available for new incoming request and cause in Agent API failures and SMSESSION management failures.
Customer uses ODBC user directory and all the ODBC connections with policy servers was going to still state or responding very slow to new incoming requests and accumulating the number of incoming agent requests causing policy server deadlock situation and all the existing connections / sessions with policy servers are broken on refresh interval.
-> Check if the Policy server connections reaching maximum by executing below command on the Policy server.
Note : Policy server connection stats are recorded in smps.log file and review the smps.log file post executing the "smpolicysrv -stats" command.
-> Check the network latency / routing table to see if the Access Gateway has problems reaching Policy server and involve the network team accordingly to fix the network issues.