JBOSS Agent is failing with the following error after upgrading to R12.1SP3.

Products

• CA Single Sign On Secure Proxy Server (SiteMinder)
• CA Single Sign On SOA Security Manager (SiteMinder)
• CA Single Sign-On

Issue/Introduction

Description:

Problem or Question:

JBOSS Agent is failing to register trusted host.

smreghost.sh returns following error.
"Failed to enable any clusters.Registration has failed.
Registration has failed."

smps.log reports handshake error.       
[CServer.cpp:1654][ERROR] Bad security handshake attempt. Handshake error: 3159 
[CServer.cpp:1659][ERROR] Handshake error: Failed to receive client hello. Client disconnected
[CServer.cpp:1827][ERROR] Failed handshake with xx.xx.xx.xx:xxx

Solution:

Troubleshooting :

You need to add debug options to the smreghost.sh script to get more information on why it is failing.

R12.1 smreghost.sh has the following command

java -classpath "$SM_SMREGHOST_CLASSPATH" com.ca.siteminder.sdk.agentapi.SmRegHost "[email protected]" 

Add the "-verbose -Dcom.ca.siteminder.sdk.agentapi.enableDebug=true" to the command as below.

java -verbose -Dcom.ca.siteminder.sdk.agentapi.enableDebug=true -classpath "$SM_SMREGHOST_CLASSPATH"
com.ca.siteminder.sdk.agentapi.SmRegHost "[email protected]" 

Then run the smreghost.sh script as below to capture the exceptions.

./smreghost.sh -i <PS IP> -u siteminder -p <password> -hn <TrustedHost name to be registered> -hc 
<HostConfigObject Name> -cf COMPAT >trace.log 2>&1 

It will generate trace.log file which would show what exceptions you are getting.

After adding debug option, you might see the following errors.

"com.ca.siteminder.sdk.agentapi.tli.a5: Error hashing client HELLO.

Caused by: java.security.NoSuchAlgorithmException: java.lang.SecurityException: Unsupported algorithm, MD5, selected for FIPS140 mode: FIPS140

Caused by: java.lang.SecurityException: Unsupported algorithm, MD5, selected for FIPS140 mode: FIPS140"

Cause :

The above error is because the JBOSS Agent was trying to register trusted host in the traditional way where it will use MD5 hash.

smreghost.sh script had the following classpath on R12.1SP3.

export SM_SMREGHOST_CLASSPATH=<JBOSS>/server/default/lib/smagentapi.jar:<JBOSS>/server/default/lib/cryptojFIPS.jar

It is referencing only the FIPS container which support FIPS routines and does not support MD5.

Solution :

Install SiteMinder SDK and copy either of the following jar files to JBOSS/lib folder and add them in the classpath for smreghost.sh

Option1 : jsafe.jar and jsafeJCE.jar files

Option2 : cryptoj.jar

Copy them to "<JBOSS>/lib" folder and "<JBOSSAGENT>/lib" folder.

Modify the smreghost.sh script as below.

From

export SM_SMREGHOST_CLASSPATH=<JBOSS>/server/default/lib/smagentapi.jar:<JBOSS>/server/default/lib/cryptojFIPS.jar

To (Option1), following content is in 1 line.

export  
SM_SMREGHOST_CLASSPATH=<JBOSS>/server/default/lib/smagentapi.jar:<JBOSS>/server/default/lib/jsafe.jar:<JBOSS>
/server/default/lib/jsafeJCE.jar:<JBOSS>/server/default/lib/cryptojFIPS.jar 

To (Option2), following content is in 1 line.

export  
SM_SMREGHOST_CLASSPATH=<JBOSS>/server/default/lib/smagentapi.jar:<JBOSS>/server/default/lib/cryptoj.jar::<JBOSS>
/server/default/lib/cryptojFIPS.jar

If you are getting different error, please capture the trace.log and submit to support for further investigation.

Environment

Release:
Component: SMJBSS