Custom SDK agent in SaS app can not connect to SiteMinder 12.8 policy server.
Article ID: 209712
Updated On: 10-10-2023
Products
Issue/Introduction
CLASSPATH="$CATALINA_BASE/lib/log4j.jar:$CATALINA_BASE/lib:$CATALINA_BASE/conf:$JRE_HOME/../lib/tools.jar:/apps/Siteminder/sdk/java/smagentapi.jar:/apps/Siteminder/sdk/java/smcrypto.jar:/apps/Siteminder/sdk/java/bc-fips-1.0.1.jar:/apps/Siteminder/sdk/java/fipsmode.jar"
After installing and configuring the SDK agent on SaS mid-tier, SDK agent can not connect to the policy server.
15:23:22.707 [main] SMTRACE: SmCluster, enable, Attempting to enable cluster id = 0
15:23:22.707 [main] SMTRACE: SmCluster, enable, Attempting to enable server index = 0
15:23:22.707 [main] SMTRACE: SmServer, enable, started
15:23:22.707 [main] SMTRACE: SmServer, createConnections, Attempt to create connections = 2
15:23:22.707 [main] SMERROR: SmServer, retryTimedoutConnections, Error retrying connection NO CONN
15:23:22.708 [main] SMTRACE: SmServer, createConnection, Currently have 0 connections to server.
15:23:22.708 [main] SMINFO: SmServer, createConnection, Creating a server connection, index = 0
15:23:22.708 [main] SMTRACE: SmServerConnection, init, Attempt to init connection
15:23:23.701 [main] SMTRACE: SmAgentTcpTransport, newInstance, Using SmAgentTcpTransport class
15:23:23.702 [main] SMTRACE: SmAgentTliSession, setup, Initiating TLI handshake
15:23:23.702 [main] SMTRACE: SmConfigAttribute, decrypt, Attempting to decrypt input = {RC2}GPVycs...................................................................A29fk
15:23:23.733 [main] SMERROR: SmServerConnection, handshake, Failed session setup.
com.ca.siteminder.sdk.agentapi.tli.SmAgentTliException: Shared secret invalid.
at com.ca.siteminder.sdk.agentapi.tli.n.br(smagentapi_obfsc:229)
at com.ca.siteminder.sdk.agentapi.connection.i.aQ(smagentapi_obfsc:328)
at com.ca.siteminder.sdk.agentapi.connection.h.aH(smagentapi_obfsc:409)
at com.ca.siteminder.sdk.agentapi.connection.h.h(smagentapi_obfsc:304)
at com.ca.siteminder.sdk.agentapi.connection.h.an(smagentapi_obfsc:235)
at com.ca.siteminder.sdk.agentapi.connection.c.an(smagentapi_obfsc:646)
Environment
Release : 12.8
Component : SITEMINDER -SDK
Cause
Handshake error is just a symptom.
The underlining error is this:
catalina.out
----------------------
Caused by: com.ca.sso.smcrypto.SmCryptoLibException: org.bouncycastle.crypto.InvalidCipherTextException: Error finalising cipher data: pad block corrupted
at com.ca.sso.smcrypto.bcfipsimpl.SmBaseCrypto.decryptBytes(SmBaseCrypto.java:421)
at com.ca.siteminder.sdk.agentapi.config.a.t(smagentapi_obfsc:342)
... 29 more
Caused by: org.bouncycastle.crypto.InvalidCipherTextException: Error finalising cipher data: pad block corrupted
at org.bouncycastle.crypto.internal.io.CipherOutputStreamImpl.close(Unknown Source)
at com.ca.sso.smcrypto.bcfipsimpl.SmBaseCrypto.decryptBytes(SmBaseCrypto.java:384)
... 30 more
Caused by: org.bouncycastle.crypto.internal.InvalidCipherTextException: pad block corrupted
at org.bouncycastle.crypto.internal.paddings.PKCS7Padding.padCount(Unknown Source)
at org.bouncycastle.crypto.internal.paddings.PaddedBufferedBlockCipher.doFinal(Unknown Source)
... 32 more"
The latest Policy Server 12.8 runs Bouncy Castle Java FIPS 1.0.1, which is a dependency change from earlier 12.7 or 12.52 release.
Any newer java SDK agent must be compiled and run with newer set of SDK jars.
Resolution
There are two problems.
1. CLASSPATH was misconfigured, missing or not loading the correct set of 12.8sp4 lib jars.
classpath should include these from SDK location :../../java/smagentapi.jar:../../java/smjavasdk2.jar:../../java/smcrypto.jar:../../java/bc-fips-1.0.1.jar, and remove legacy jar like fipsmode.jar
2. Customer has regular agent and SDK agent on the same server. They registered SDK agent with regular agent command and regular agent lib path, which causes the error.
Regular agent is unlike SDK agent, it uses C based code, hence regular agent is not impacted with Java jar change. SmHost.conf file created with regular agent can NOT be used with SDK agent.
When using SDK agent, one must follow these steps in the doc to do SmRegHost.
Windows platform:
SM_SMREGHOST_CLASSPATH="c:\ca\sdk\java\smagentapi.jar;c:\ca\sdk\java\smcrypto.jar;c:\ca\sdk\java\bc-fips-1.0.1.jar"
java -Dcom.ca.siteminder.sdk.agentapi.enableDebug="true" -classpath %SM_SMREGHOST_CLASSPATH% com.ca.siteminder.sdk.agentapi.SmRegHost -i 10.0.0.1 -hc host_conf1 -hn trustedhost3 -u <super_user>-p <superuser_password> -f "c:\ca\sdk\SmHost.conf"
Please alter above command to Linux platform before direct usage.
Linux platform:
SM_SMREGHOST_CLASSPATH=/opt/CA/sdk/java/smagentapi.jar:/opt/CA/sdk/java/smcrypto.jar:/opt/CA/sdk/java/bc-fips-1.0.1.jar
If policy server is in COMPAT mode.
java -Dcom.ca.siteminder.sdk.agentapi.enableDebug="true" -classpath $SM_SMREGHOST_CLASSPATH com.ca.siteminder.sdk.agentapi.SmRegHost -i 10.0.0.1 -hc host_conf1 -hn trustedhost3 -u <super_user>-p <superuser_password>
If policy server is in FIPS ONLY mode.
java -Dcom.ca.siteminder.sdk.agentapi.enableDebug="true" -classpath $SM_SMREGHOST_CLASSPATH com.ca.siteminder.sdk.agentapi.SmRegHost -i 10.0.0.1 -hc host_conf1 -hn trustedhost3 -u <super_user>-p <superuser_password> -cf ONLY
-cf option: <Crypto FIPS140 mode (COMPAT or MIGRATE or ONLY)>
After generated new SmHost.conf and corrected CLASSPATH, SDK agent is able to connect.
Additional Information
Feedback
