ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

Error : Web Agent reports Failover from cluster [0] to cluster [1]

book

Article ID: 141422

calendar_today

Updated On:

Products

CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) SITEMINDER CA Single Sign On Agents (SiteMinder)

Issue/Introduction

 

When running a Siteminder Web Agent and when the user tries to access

   https://myserver.mydomain.com/myweb/mypage.jsp

then the browser receives return code 500 with the mention :

   Internal Server Error

   The server encountered an internal error or misconfiguration and
   was unable to complete your request.

   Please contact the server administrator at [email protected] to
   inform them of the time this error occurred, and the actions you
   performed just before this error.

   More information about this error may be available in the server
   error log.

   Additionally, a 500 Internal Server Error error was encountered
   while trying to use an Error Document to handle the request.

The Web Agent reports line :

    Failover from cluster [0] to cluster [1]

 

Cause

 

Applying the following template to the Web Agent traces config file :

components: AgentFramework, HTTPAgent, WebAgent, AgentFunc, Agent_Con_Manager
data: Date, Time, PreciseTime, Pid, Tid, TransactionID, Message, SrcFile, Function, RequestIPAddr, IPAddr, IPPort, AgentName, Domain, DomainOID, Realm, RealmOID, Resource, Action, User, SessionSpec, SessionID, CertSerial, SubjectDN, IssuerDN, UserDN, Threshold, Throughput, MinThroughput, MaxThroughput, HandleCount, BusyHandleCount, FreeHandleCount, State, ClusterID

The Web Agent thread shows that it losts connection with the Policy
Server as it waits too long to get the response back at the network
level.

WebAgent.log

  [21557/2734651136][Thu Nov 28 2019 08:25:31][SmAgentAPI.cpp:315][INFO]
  [sm-AgentConMgr-00050] Failover from cluster [0] to cluster [1].

[...]

  [21557/2766120704][Thu Nov 28 2019 08:26:38][CSmLowLevelAgent.cpp:546]
  [ERROR][sm-AgentFramework-00520] LLA: SiteMinder Agent Api function failed -
  'Sm_AgentApi_IsProtectedEx' returned '-2'.

[...]

  [21557/2703181568][Thu Nov 28 2019 08:26:57][CSmLowLevelAgent.cpp:546]
  [ERROR][sm-AgentFramework-00520] LLA: SiteMinder Agent Api function failed -
  'Sm_AgentApi_IsProtectedEx' returned '-1'.
  WebAgentTrace.log 

  [11/28/2019][08:26:38][08:26:38.830][21557][2766120704][]
  ["SmAgentTli_TCPReceiveBuf returns TIME-OUT. No connection retry."]
  [SmClient.cpp:1673][CSmServerHandle::ProcessRequest][][10.1.1.0]
  [44443][][][][][][][][][][][][][][][][][][][3][][][ACTIVE][]

  [11/28/2019][08:26:38][08:26:38.830][21557][2766120704][]
  [Request failed with 2 error code.][SmClient.cpp:2745]
  [ProcessRequest][][][][][][][][][][][][][][][][][][][][][]
  [4][][][ACTIVE][9]

  [11/28/2019][08:26:38][08:26:38.830][21557][2766120704][]
  [Server State transition from ACTIVE to INTER][SmClient.cpp:1810]
  [CSmServerHandle::StateTransition][][10.0.0.1][44443][][][][][][]
  [][][][][][][][][][][][][3][][][INTER][]

  [11/28/2019][08:26:38][08:26:38.830][21557][2766120704][]
  [Request failed with 2 error code.][SmClient.cpp:2762][ProcessRequest]
  [][][][][][][][][][][][][][][][][][][][][][4][][][ACTIVE][9]

  [11/28/2019][08:26:38][08:26:38.830][21557][2766120704][]
  [Server State transition from INTER to INACTIVE][SmClient.cpp:1810]
  [CSmServerHandle::StateTransition][][10.0.0.1][44443][][][][][][][]
  [][][][][][][][][][][][3][][][INACTIVE][]

  [11/28/2019][08:26:38][08:26:38.830][21557][2766120704][]
  [The number of active servers fell below the threshold.]
  [SmClient.cpp:2846][ProcessRequest][][][][][][][][][][][][][][][]
  [][][][][][][4][][][ACTIVE][9]

  [11/28/2019][08:26:38][08:26:38.830][21557][2766120704][]
  [Cluster State transition from ACTIVE to INACTIVE][SmClient.cpp:3273]
  [CSmCluster::StateTransition][][][][][][][][][][][][][][][][][][][][]
  [][4][][][INACTIVE][9]

  [11/28/2019][08:26:38][08:26:38.830][21557][2766120704][]
  [No active clusters found.][SmAgentAPI.cpp:521][ProcessRequest]
  [][][][][][][][][][][][][][][][][][][][][][][][][][]

  [11/28/2019][08:26:38][08:26:38.830][21557][2766120704][]
  [Leave function ProcessRequest][SmAgentAPI.cpp:527][ProcessRequest]
  [][][][][][][][][][][][][][][][][][][][][][][][][][]

  [11/28/2019][08:26:38][08:26:38.830][21557][2766120704][]
  [Leave function Sm_AgentApi_IsProtected][SmAgentAPI.cpp:1813]
  [Sm_AgentApi_IsProtected][][][][][][][][][][][][][][][][][][][][][]
  [][][][][]

  [11/28/2019][08:26:38][08:26:38.830][21557][2766120704]
  [00000000000000000000000064f45899-5435-5ddf7672-a4dfa700-2cd73432c50]
  [Communication failure between SiteMinder policy server and web agent.]
  [CSmLowLevelAgent.cpp:541][IsResourceProtected][10.0.0.2][*10.0.0.3][]
  [myagent][][][][][/myweb/mypage.jsp][GET][DOMAIN\MYUSER]
  [k3ZVtGzXIMnuIZ1qHzR7BxzYMbzvuwL8PLJH

  [...]

  qdpI=][][][][CN=myname,DC=mydomain,DC=com][][][][][][][][][]

Environment

 

Web Agent 12.52SP1CR11 on Apache 2.4.46 on RedHat 6
Policy Server 12.8 on Redhat 8

 

Resolution

 

- Investigate network, load balancer and firewall and make sure that
  there's no timeout on the TCP Protocol. If there's one, make sure
  that the TCP Protocol timeout is big enough.

- On the Web Agent and Policy Server, enable the environment variable
  SM_ENABLE_TCP_KEEPALIVE to insure that both component won't try to
  use a connection that has been terminated on the firewall or
  loadbalancer :

  SM_ENABLE_TCP_KEEPALIVE (1)

 

Additional Information

 

(1)

    Web Agent and Policy Server Network Communication Disruption
    https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=42108