Error: Web Agent reports Failover from cluster [0] to cluster [1]
search cancel

Error: Web Agent reports Failover from cluster [0] to cluster [1]

book

Article ID: 141422

calendar_today

Updated On:

Products

CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) SITEMINDER CA Single Sign On Agents (SiteMinder)

Issue/Introduction

 

When running a Siteminder Web Agent and when the user tries to access

   https://myserver.example.com/myweb/mypage.jsp

then the browser receives return code 500 with the mention:

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at [email protected] to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Additionally, a 500 Internal Server Error error was encountered while trying to use an Error Document to handle the request.

The Web Agent reports line:

    Failover from cluster [0] to cluster [1]

 

Environment

 

Web Agent 12.52SP1CR11 on Apache 2.4.46 on RedHat 6
Policy Server 12.8 on Redhat 8

 

Cause

 

Applying the following template to the Web Agent traces config file:

components: AgentFramework, HTTPAgent, WebAgent, AgentFunc, Agent_Con_Manager
data: Date, Time, PreciseTime, Pid, Tid, TransactionID, Message, SrcFile, Function, RequestIPAddr, IPAddr, IPPort, AgentName, Domain, DomainOID, Realm, RealmOID, Resource, Action, User, SessionSpec, SessionID, CertSerial, SubjectDN, IssuerDN, UserDN, Threshold, Throughput, MinThroughput, MaxThroughput, HandleCount, BusyHandleCount, FreeHandleCount, State, ClusterID

The Web Agent thread shows that it lost connection with the Policy Server as it waits too long to get the response back at the network level.

WebAgent.log

[21557/2734651136][Thu Nov 28 2019 08:25:31][SmAgentAPI.cpp:315][INFO][sm-AgentConMgr-00050] Failover from cluster [0] to cluster [1].

[...]

[21557/2766120704][Thu Nov 28 2019 08:26:38][CSmLowLevelAgent.cpp:546][ERROR][sm-AgentFramework-00520] LLA: SiteMinder Agent Api function failed - 'Sm_AgentApi_IsProtectedEx' returned '-2'.

[...]

[21557/2703181568][Thu Nov 28 2019 08:26:57][CSmLowLevelAgent.cpp:546][ERROR][sm-AgentFramework-00520] LLA: SiteMinder Agent Api function failed - 'Sm_AgentApi_IsProtectedEx' returned '-1'.

 

WebAgentTrace.log 

[11/28/2019][08:26:38][08:26:38.830][21557][2766120704][]["SmAgentTli_TCPReceiveBuf returns TIME-OUT. No connection retry."][SmClient.cpp:1673][CSmServerHandle::ProcessRequest][][10.0.0.1][44443][][][][][][][][][][][][][][][][][][][3][][][ACTIVE][]

[11/28/2019][08:26:38][08:26:38.830][21557][2766120704][][Request failed with 2 error code.][SmClient.cpp:2745][ProcessRequest][][][][][][][][][][][][][][][][][][][][][][4][][][ACTIVE][9]

[11/28/2019][08:26:38][08:26:38.830][21557][2766120704][][Server State transition from ACTIVE to INTER][SmClient.cpp:1810][CSmServerHandle::StateTransition][][10.0.0.1][44443][][][][][][][][][][][][][][][][][][][3][][][INTER][]

[11/28/2019][08:26:38][08:26:38.830][21557][2766120704][][Request failed with 2 error code.][SmClient.cpp:2762][ProcessRequest][][][][][][][][][][][][][][][][][][][][][][4][][][ACTIVE][9]

[11/28/2019][08:26:38][08:26:38.830][21557][2766120704][][Server State transition from INTER to INACTIVE][SmClient.cpp:1810][CSmServerHandle::StateTransition][][10.0.0.1][44443][][][][][][][][][][][][][][][][][][][3][][][INACTIVE][]

[11/28/2019][08:26:38][08:26:38.830][21557][2766120704][][The number of active servers fell below the threshold.][SmClient.cpp:2846][ProcessRequest][][][][][][][][][][][][][][][][][][][][][][4][][][ACTIVE][9]

[11/28/2019][08:26:38][08:26:38.830][21557][2766120704][][Cluster State transition from ACTIVE to INACTIVE][SmClient.cpp:3273][CSmCluster::StateTransition][][][][][][][][][][][][][][][][][][][][][][4][][][INACTIVE][9]

[11/28/2019][08:26:38][08:26:38.830][21557][2766120704][][No active clusters found.][SmAgentAPI.cpp:521][ProcessRequest][][][][][][][][][][][][][][][][][][][][][][][][][][]

[11/28/2019][08:26:38][08:26:38.830][21557][2766120704][][Leave function ProcessRequest][SmAgentAPI.cpp:527][ProcessRequest][][][][][][][][][][][][][][][][][][][][][][][][][][]

[11/28/2019][08:26:38][08:26:38.830][21557][2766120704][][Leave function Sm_AgentApi_IsProtected][SmAgentAPI.cpp:1813][Sm_AgentApi_IsProtected][][][][][][][][][][][][][][][][][][][][][][][][][][]

[11/28/2019][08:26:38][08:26:38.830][21557][2766120704][00000000000000000000000064f45899-5435-5ddf7672-**************][Communication failure between SiteMinder policy server and web agent.][CSmLowLevelAgent.cpp:541][IsResourceProtected][10.0.0.2][*10.0.0.3][][myagent][][][][][/myweb/mypage.jsp][GET][DOMAIN\MYUSER][k3ZVtGzXIMnuIZ1qHzR7BxzYMbzvuwL8PLJH [...] qdpI=][][][][CN=myname,DC=mydomain,DC=com][][][][][][][][][]

 

Resolution

 

  • Investigate the network, load balancer, and firewall and make sure that there's no timeout on the TCP Protocol. If there's one, make sure that the TCP Protocol timeout is big enough.
  • On the Web Agent and Policy Server, enable the environment variable SM_ENABLE_TCP_KEEPALIVE to ensure that both components won't try to use a connection that has been terminated on the firewall or load balancer:

    SM_ENABLE_TCP_KEEPALIVE (1)

Additional Information

 

(1)

    Error: 500 Web Agent and Policy Server network communication problem
    

Powered by Wolken Software