Custom SDK agent in Sas app can not connect to SiteMinder 12.8SP4 policy server.

book

Article ID: 209712

calendar_today

Updated On:

Products

SITEMINDER

Issue/Introduction

OS: Red Hat Enterprise Linux Server release 7.8
Policy Server Version: 12.80.400.2278
Application: SAS Web appserver on Linux as well.

CLASSPATH="$CATALINA_BASE/lib/log4j.jar:$CATALINA_BASE/lib:$CATALINA_BASE/conf:$JRE_HOME/../lib/tools.jar:/apps/Siteminder/sdk/java/smagentapi.jar:/apps/Siteminder/sdk/java/smcrypto.jar:/apps/Siteminder/sdk/java/bc-fips-1.0.1.jar:/apps/Siteminder/sdk/java/fipsmode.jar"

After installing and configuring the SDK agent on SAS mid-tier, SDK agent can not connect to policy server.

15:23:22.707 [main] SMTRACE: SmCluster, enable, Attempting to enable cluster id = 0
15:23:22.707 [main] SMTRACE: SmCluster, enable, Attempting to enable server index = 0
15:23:22.707 [main] SMTRACE: SmServer, enable, started
15:23:22.707 [main] SMTRACE: SmServer, createConnections, Attempt to create connections = 2
15:23:22.707 [main] SMERROR: SmServer, retryTimedoutConnections, Error retrying connection NO CONN
15:23:22.708 [main] SMTRACE: SmServer, createConnection, Currently have 0 connections to server.
15:23:22.708 [main] SMINFO: SmServer, createConnection, Creating a server connection, index = 0
15:23:22.708 [main] SMTRACE: SmServerConnection, init, Attempt to init connection
15:23:23.701 [main] SMTRACE: SmAgentTcpTransport, newInstance, Using SmAgentTcpTransport class
15:23:23.702 [main] SMTRACE: SmAgentTliSession, setup, Initiating TLI handshake
15:23:23.702 [main] SMTRACE: SmConfigAttribute, decrypt, Attempting to decrypt input = {RC2}GPVycsczZW/hvg37UxNnmoI9afWwnXP0RcsBsf7iZ9.............gSRY8oHgYjLUxJPIXIgSKGA29fk
15:23:23.733 [main] SMERROR: SmServerConnection, handshake, Failed session setup.
com.ca.siteminder.sdk.agentapi.tli.SmAgentTliException: Shared secret invalid.
        at com.ca.siteminder.sdk.agentapi.tli.n.br(smagentapi_obfsc:229)
        at com.ca.siteminder.sdk.agentapi.connection.i.aQ(smagentapi_obfsc:328)
        at com.ca.siteminder.sdk.agentapi.connection.h.aH(smagentapi_obfsc:409)
        at com.ca.siteminder.sdk.agentapi.connection.h.h(smagentapi_obfsc:304)
        at com.ca.siteminder.sdk.agentapi.connection.h.an(smagentapi_obfsc:235)
        at com.ca.siteminder.sdk.agentapi.connection.c.an(smagentapi_obfsc:646)

Cause

Handshake error is just a symptom.

The underlining error is this:

catalina.out
----------------------
Caused by: com.ca.sso.smcrypto.SmCryptoLibException: org.bouncycastle.crypto.InvalidCipherTextException: Error finalising cipher data: pad block corrupted
 at com.ca.sso.smcrypto.bcfipsimpl.SmBaseCrypto.decryptBytes(SmBaseCrypto.java:421)
 at com.ca.siteminder.sdk.agentapi.config.a.t(smagentapi_obfsc:342)
 ... 29 more
Caused by: org.bouncycastle.crypto.InvalidCipherTextException: Error finalising cipher data: pad block corrupted
 at org.bouncycastle.crypto.internal.io.CipherOutputStreamImpl.close(Unknown Source)
 at com.ca.sso.smcrypto.bcfipsimpl.SmBaseCrypto.decryptBytes(SmBaseCrypto.java:384)
 ... 30 more
Caused by: org.bouncycastle.crypto.internal.InvalidCipherTextException: pad block corrupted
 at org.bouncycastle.crypto.internal.paddings.PKCS7Padding.padCount(Unknown Source)
 at org.bouncycastle.crypto.internal.paddings.PaddedBufferedBlockCipher.doFinal(Unknown Source)
 ... 32 more"

The latest Policy Server 12.8 runs Bouncy Castle Java FIPS 1.0.1, which is a dependency change from earlier 12.7 or 12.52 release.

Any newer java SDK agent must be compiled and run with newer set of SDK jars.

Environment

Release : 12.8

Component : SITEMINDER -SDK

Resolution

There are two problems.

1. CLASSPATH was misconfigured, missing or not loading the correct set of 12.8sp4 lib jars.

classpath should include these from SDK location :../../java/smagentapi.jar:../../java/smjavasdk2.jar:../../java/smcrypto.jar:../../java/bc-fips-1.0.1.jar, and remove legacy jar like fipsmode.jar

2. Customer has regular agent and SDK agent on the same server. They registered SDK agent with regular agent command and regular agent lib path, which causes the error.

When using SDK agent, one must follow these steps in the doc to do SmRegHost, 

SM_SMREGHOST_CLASSPATH="c:\ca\sdk\java\smagentapi.jar;c:\ca\sdk\java\smcrypto.jar;c:\ca\sdk\java\bc-fips-1.0.1.jar"
java -Dcom.ca.siteminder.sdk.agentapi.enableDebug="true" -classpath %SM_SMREGHOST_CLASSPATH% com.ca.siteminder.sdk.agentapi.SmRegHost -i 127.0.0.1 -hc host_conf1 -hn trustedhost3 -u siteminder -p password -f  "c:\ca\sdk\SmHost.conf"

Please alter above command to Linux platform before direct usage.

After generated new SmHost.conf and corrected CLASSPATH, SDK agent is able to connect.

Additional Information

https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/programming/sdks/programming-in-java/agent-api-in-java.html