Description:
Problem or Question:
JBOSS Agent is failing to register trusted host.
smreghost.sh returns following error.
"Failed to enable any clusters.Registration has failed.
Registration has failed."
smps.log reports handshake error. [CServer.cpp:1654][ERROR] Bad security handshake attempt. Handshake error: 3159 [CServer.cpp:1659][ERROR] Handshake error: Failed to receive client hello. Client disconnected [CServer.cpp:1827][ERROR] Failed handshake with xx.xx.xx.xx:xxx
Solution:
Troubleshooting :
You need to add debug options to the smreghost.sh script to get more information on why it is failing.
R12.1 smreghost.sh has the following command
java -classpath "$SM_SMREGHOST_CLASSPATH" com.ca.siteminder.sdk.agentapi.SmRegHost "[email protected]"
Add the "-verbose -Dcom.ca.siteminder.sdk.agentapi.enableDebug=true" to the command as below.
java -verbose -Dcom.ca.siteminder.sdk.agentapi.enableDebug=true -classpath "$SM_SMREGHOST_CLASSPATH" com.ca.siteminder.sdk.agentapi.SmRegHost "[email protected]"
Then run the smreghost.sh script as below to capture the exceptions.
./smreghost.sh -i <PS IP> -u siteminder -p <password> -hn <TrustedHost name to be registered> -hc <HostConfigObject Name> -cf COMPAT >trace.log 2>&1
It will generate trace.log file which would show what exceptions you are getting.
After adding debug option, you might see the following errors.
"com.ca.siteminder.sdk.agentapi.tli.a5: Error hashing client HELLO.
Caused by: java.security.NoSuchAlgorithmException: java.lang.SecurityException: Unsupported algorithm, MD5, selected for FIPS140 mode: FIPS140
Caused by: java.lang.SecurityException: Unsupported algorithm, MD5, selected for FIPS140 mode: FIPS140"
Cause :
The above error is because the JBOSS Agent was trying to register trusted host in the traditional way where it will use MD5 hash.
smreghost.sh script had the following classpath on R12.1SP3.
export SM_SMREGHOST_CLASSPATH=<JBOSS>/server/default/lib/smagentapi.jar:<JBOSS>/server/default/lib/cryptojFIPS.jar
It is referencing only the FIPS container which support FIPS routines and does not support MD5.
Solution :
Install SiteMinder SDK and copy either of the following jar files to JBOSS/lib folder and add them in the classpath for smreghost.sh
Option1 : jsafe.jar and jsafeJCE.jar files
Option2 : cryptoj.jar
Copy them to "<JBOSS>/lib" folder and "<JBOSSAGENT>/lib" folder.
Modify the smreghost.sh script as below.
From
export SM_SMREGHOST_CLASSPATH=<JBOSS>/server/default/lib/smagentapi.jar:<JBOSS>/server/default/lib/cryptojFIPS.jar
To (Option1), following content is in 1 line.
export SM_SMREGHOST_CLASSPATH=<JBOSS>/server/default/lib/smagentapi.jar:<JBOSS>/server/default/lib/jsafe.jar:<JBOSS> /server/default/lib/jsafeJCE.jar:<JBOSS>/server/default/lib/cryptojFIPS.jar
To (Option2), following content is in 1 line.
export SM_SMREGHOST_CLASSPATH=<JBOSS>/server/default/lib/smagentapi.jar:<JBOSS>/server/default/lib/cryptoj.jar::<JBOSS> /server/default/lib/cryptojFIPS.jar
If you are getting different error, please capture the trace.log and submit to support for further investigation.