Custom SDK agent in SaS app can not connect to SiteMinder 12.8 policy server.
search cancel

Custom SDK agent in SaS app can not connect to SiteMinder 12.8 policy server.


Article ID: 209712


Updated On:




OS: Red Hat Enterprise Linux Server release 7.8
Policy Server Version: 12.80.400.2278
Application: SaS web app server on Linux as well.


After installing and configuring the SDK agent on SaS mid-tier, SDK agent can not connect to the policy server.

15:23:22.707 [main] SMTRACE: SmCluster, enable, Attempting to enable cluster id = 0
15:23:22.707 [main] SMTRACE: SmCluster, enable, Attempting to enable server index = 0
15:23:22.707 [main] SMTRACE: SmServer, enable, started
15:23:22.707 [main] SMTRACE: SmServer, createConnections, Attempt to create connections = 2
15:23:22.707 [main] SMERROR: SmServer, retryTimedoutConnections, Error retrying connection NO CONN
15:23:22.708 [main] SMTRACE: SmServer, createConnection, Currently have 0 connections to server.
15:23:22.708 [main] SMINFO: SmServer, createConnection, Creating a server connection, index = 0
15:23:22.708 [main] SMTRACE: SmServerConnection, init, Attempt to init connection
15:23:23.701 [main] SMTRACE: SmAgentTcpTransport, newInstance, Using SmAgentTcpTransport class
15:23:23.702 [main] SMTRACE: SmAgentTliSession, setup, Initiating TLI handshake
15:23:23.702 [main] SMTRACE: SmConfigAttribute, decrypt, Attempting to decrypt input = {RC2}GPVycs...................................................................A29fk
15:23:23.733 [main] SMERROR: SmServerConnection, handshake, Failed session setup. Shared secret invalid.


Release : 12.8



Handshake error is just a symptom.

The underlining error is this:

Caused by: org.bouncycastle.crypto.InvalidCipherTextException: Error finalising cipher data: pad block corrupted
 ... 29 more
Caused by: org.bouncycastle.crypto.InvalidCipherTextException: Error finalising cipher data: pad block corrupted
 at Source)
 ... 30 more
Caused by: org.bouncycastle.crypto.internal.InvalidCipherTextException: pad block corrupted
 at org.bouncycastle.crypto.internal.paddings.PKCS7Padding.padCount(Unknown Source)
 at org.bouncycastle.crypto.internal.paddings.PaddedBufferedBlockCipher.doFinal(Unknown Source)
 ... 32 more"

The latest Policy Server 12.8 runs Bouncy Castle Java FIPS 1.0.1, which is a dependency change from earlier 12.7 or 12.52 release.

Any newer java SDK agent must be compiled and run with newer set of SDK jars.


There are two problems.

1. CLASSPATH was misconfigured, missing or not loading the correct set of 12.8sp4 lib jars.

classpath should include these from SDK location :../../java/smagentapi.jar:../../java/smjavasdk2.jar:../../java/smcrypto.jar:../../java/bc-fips-1.0.1.jar, and remove legacy jar like fipsmode.jar

2. Customer has regular agent and SDK agent on the same server. They registered SDK agent with regular agent command and regular agent lib path, which causes the error.

Regular agent is unlike SDK agent, it uses C based code, hence regular agent is not impacted with Java jar change. SmHost.conf file created with regular agent can NOT be used with SDK agent.

When using SDK agent, one must follow these steps in the doc to do SmRegHost.

Windows platform:

java"true" -classpath %SM_SMREGHOST_CLASSPATH% -i -hc host_conf1 -hn trustedhost3 -u <super_user>-p <superuser_password> -f  "c:\ca\sdk\SmHost.conf"

Please alter above command to Linux platform before direct usage.

Linux platform:


If policy server is in COMPAT mode.
java"true" -classpath $SM_SMREGHOST_CLASSPATH -i -hc host_conf1 -hn trustedhost3 -u <super_user>-p <superuser_password>

If policy server is in FIPS ONLY mode.
java"true" -classpath $SM_SMREGHOST_CLASSPATH -i -hc host_conf1 -hn trustedhost3 -u <super_user>-p <superuser_password> -cf ONLY

-cf option: <Crypto FIPS140 mode (COMPAT or MIGRATE or ONLY)>

After generated new SmHost.conf and corrected CLASSPATH, SDK agent is able to connect.

Additional Information