All application's URLs down - not accessible and reachable - Web Agent
search cancel

All application's URLs down - not accessible and reachable - Web Agent

book

Article ID: 258456

calendar_today

Updated On:

Products

SITEMINDER CA Single Sign On Agents (SiteMinder) CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign On Federation (SiteMinder)

Issue/Introduction

 

Running Web Agent and CA Access Gateway (SPS), for an hour, all URLs were down and not accessible.

The Policy Server smps.log reports socket errors, which point to network problems too (1)(2).

 

Cause

 

10 Policy Servers are defined in the HCO:

webagent.log:

[81698/4119291648][Mon Dec 12 2022 08:07:55] SiteMinder Agent API Host Configuration: 
[81698/4119291648][Mon Dec 12 2022 08:07:55] enablefailover='NO'.
[81698/4119291648][Mon Dec 12 2022 08:07:55] policyserver='10.0.0.4,44441,44442,44443'.
[81698/4119291648][Mon Dec 12 2022 08:07:55] policyserver='10.0.0.5,44441,44442,44443'.
[81698/4119291648][Mon Dec 12 2022 08:07:55] policyserver='10.0.0.6,44441,44442,44443'.
[81698/4119291648][Mon Dec 12 2022 08:07:55] policyserver='10.0.0.7,44441,44442,44443'.
[81698/4119291648][Mon Dec 12 2022 08:07:55] policyserver='10.0.0.8,44441,44442,44443'.
[81698/4119291648][Mon Dec 12 2022 08:07:55] policyserver='10.0.0.9,44441,44442,44443'.
[81698/4119291648][Mon Dec 12 2022 08:07:55] policyserver='10.0.0.10,44441,44442,44443'.
[81698/4119291648][Mon Dec 12 2022 08:07:55] policyserver='10.0.0.11,44441,44442,44443'.
[81698/4119291648][Mon Dec 12 2022 08:07:55] policyserver='10.0.0.12,44441,44442,44443'.
[81698/4119291648][Mon Dec 12 2022 08:07:55] policyserver='10.0.0.13,44441,44442,44443'.

Web Agent reports a problem to connect to the Policy Servers defined in the HCO around 06:50:48:

error.log:

[23/Dec/2022:06:50:48] [Error] SiteMinder Agent
        Unable to load SiteMinder host configuration object or host configuration file.
        /opt/CA/webagent/config/SmHost.conf
        06 00 00 00 

[...omitted for brevity...]

[23/Dec/2022:07:18:22] [Error] SiteMinder Agent
        Unable to load SiteMinder host configuration object or host configuration file.
        /opt/CA/webagent/config/SmHost.conf
        06 00 00 00 

[...omitted for brevity...]

[23/Dec/2022:08:33:06] [Info] [CA WebAgent IPC] [212952] [CSmSem::SemRm] Removed semaphore ######
[23/Dec/2022:08:33:09] [Info] [CA WebAgent IPC] [217338] [CSmSem::SemRm] Already removed semaphore ######
[23/Dec/2022:08:33:09] [Info] [CA WebAgent IPC] [209303] [CSmSem::SemRm] Already removed semaphore ######

The Policy Server reports socket errors 104 and 107: 

smps.log:

[21919/139910958798592][Fri Dec 23 2022 06:56:59][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107
[21919/139910958798592][Fri Dec 23 2022 07:08:17][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107
[21919/139911311128320][Fri Dec 23 2022 07:12:55][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139911589086976][Fri Dec 23 2022 07:12:55][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139910958798592][Fri Dec 23 2022 07:12:55][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107
[21919/139910958798592][Fri Dec 23 2022 07:12:55][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107
[21919/139910958798592][Fri Dec 23 2022 07:12:55][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107
[21919/139911614265088][Fri Dec 23 2022 07:12:55][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139911597479680][Fri Dec 23 2022 07:12:55][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139911589086976][Fri Dec 23 2022 07:12:56][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139911614265088][Fri Dec 23 2022 07:12:56][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139911614265088][Fri Dec 23 2022 07:12:56][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139910958798592][Fri Dec 23 2022 07:12:56][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107
[21919/139911605872384][Fri Dec 23 2022 07:12:56][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139911605872384][Fri Dec 23 2022 07:12:57][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139910958798592][Fri Dec 23 2022 07:12:57][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107
[21919/139910958798592][Fri Dec 23 2022 07:12:57][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107
[21919/139911597479680][Fri Dec 23 2022 07:12:57][CServer.cpp:2121][ERROR][sm-Tunnel-00010] Bad security handshake attempt. Handshake error: 3152
[21919/139911597479680][Fri Dec 23 2022 07:12:57][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139911614265088][Fri Dec 23 2022 07:12:57][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139910958798592][Fri Dec 23 2022 07:12:58][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107
[21919/139911589086976][Fri Dec 23 2022 07:12:58][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139910958798592][Fri Dec 23 2022 07:12:58][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107
[21919/139910958798592][Fri Dec 23 2022 07:12:58][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107
[21919/139911614265088][Fri Dec 23 2022 07:12:58][CServer.cpp:2128][ERROR][sm-Tunnel-00030] Handshake error: Failed to receive client hello. Socket error 104
[21919/139910958798592][Fri Dec 23 2022 07:12:58][CServer.cpp:1814][ERROR][sm-Server-01050] Failed to initialize TCP client connection. Socket error 107

and these errors appear after the outage time too, but less often.

 

Resolution

 

  • Investigate network disruption and traffic at the time of the issue with the network team;
  • On the Web Agent and the Policy Server, implement the SM_ENABLE_TCP_KEEPALIVE (3);
  • On the Web Agent, implement the agentwaittime (4);

 

Additional Information

 

(1)

    Errors: ldap 81/48 - Socket error 107 - Error code 2 - Handshake 3154
    

(2)
  
    Error: Failed to receive client hello. Socket error 104 in smps.log
    

(3)

    Policy Server Hangs after Web Agent Communication Failure
    

(4)

    AgentWaitTime parameter Explained in webagent.conf
    

Powered by Wolken Software