search cancel

JBOSS Agent is failing with the following error after upgrading to R12.1SP3.


Article ID: 49711


Updated On:


CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign-On



Problem or Question:

JBOSS Agent is failing to register trusted host. returns following error.
"Failed to enable any clusters.Registration has failed.
Registration has failed."

smps.log reports handshake error.       
[CServer.cpp:1654][ERROR] Bad security handshake attempt. Handshake error: 3159 
[CServer.cpp:1659][ERROR] Handshake error: Failed to receive client hello. Client disconnected
[CServer.cpp:1827][ERROR] Failed handshake with xx.xx.xx.xx:xxx


Troubleshooting :

You need to add debug options to the script to get more information on why it is failing.

R12.1 has the following command

java -classpath "$SM_SMREGHOST_CLASSPATH" "[email protected]" 

Add the "-verbose" to the command as below.

java -verbose -classpath "$SM_SMREGHOST_CLASSPATH" "[email protected]" 

Then run the script as below to capture the exceptions.

./ -i <PS IP> -u siteminder -p <password> -hn <TrustedHost name to be registered> -hc 
<HostConfigObject Name> -cf COMPAT >trace.log 2>&1 

It will generate trace.log file which would show what exceptions you are getting.

After adding debug option, you might see the following errors.

" Error hashing client HELLO.

Caused by: java.lang.SecurityException: Unsupported algorithm, MD5, selected for FIPS140 mode: FIPS140

Caused by: java.lang.SecurityException: Unsupported algorithm, MD5, selected for FIPS140 mode: FIPS140"

Cause :

The above error is because the JBOSS Agent was trying to register trusted host in the traditional way where it will use MD5 hash. script had the following classpath on R12.1SP3.

export SM_SMREGHOST_CLASSPATH=<JBOSS>/server/default/lib/smagentapi.jar:<JBOSS>/server/default/lib/cryptojFIPS.jar

It is referencing only the FIPS container which support FIPS routines and does not support MD5.

Solution :

Install SiteMinder SDK and copy either of the following jar files to JBOSS/lib folder and add them in the classpath for

Option1 : jsafe.jar and jsafeJCE.jar files

Option2 : cryptoj.jar

Copy them to "<JBOSS>/lib" folder and "<JBOSSAGENT>/lib" folder.

Modify the script as below.


export SM_SMREGHOST_CLASSPATH=<JBOSS>/server/default/lib/smagentapi.jar:<JBOSS>/server/default/lib/cryptojFIPS.jar

To (Option1), following content is in 1 line.


To (Option2), following content is in 1 line.


If you are getting different error, please capture the trace.log and submit to support for further investigation.


Component: SMJBSS