Unable to start application protected by a custom agent in Weblogic.

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• CA Single Sign-On

Issue/Introduction

When starting the Weblogic server, we are getting the following when it initialized the custom agent embedded in a weblogic server :

> Error message Initialisation failed for SECURITY_MANAGER : netegrity/siteminder/javaagent/ServiceSession 

 

Environment

PS : 12.52 SP1 on Solaris sparc 10SDK : r12.52SP1Weblogic : 10.3.6 on Solaris sparc 10

Cause

PS is configured in FIPS only mode.

 

Sample of smps.log :

[17559/1][Tue Jun 20 2017 08:21:58][CServer.cpp:4006][INFO][sm-Server-04450] Policy Server employing only FIPS-140 cryptographic algorithms.

 

When Agent tries to initialize we can see the following:

[17559/10][Wed Jun 21 2017 10:17:55][CServer.cpp:1965][ERROR][sm-Tunnel-00010] Bad security handshake attempt. Handshake error: 3153

[17559/10][Wed Jun 21 2017 10:17:55][CServer.cpp:1974][ERROR][sm-Tunnel-00040] Handshake error: Bad version number or FIPS mode in hello message

[17559/10][Wed Jun 21 2017 10:17:55][CServer.cpp:2137][ERROR][sm-Server-01070] Failed handshake with 192.168.200.76:10190 

Resolution

If you are using 4x connection, you have to set the following environment variable to the account which is starting WebLogic server : 

 export CA_SM_PS_FIPS140=ONLY 

 

If you are using 5x connection review the configuration of the SmHost.conf file according to documentation : 

https://docops.ca.com/ca-single-sign-on/12-52-sp1/en/upgrading/using-fips-compliant-algorithms/how-to-configure-fips-only-mode