ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

Custom SDK agent in Sas app can not connect to SiteMinder 12.8SP4 policy server.

book

Article ID: 209712

calendar_today

Updated On:

Products

SITEMINDER

Issue/Introduction

OS: Red Hat Enterprise Linux Server release 7.8
Policy Server Version: 12.80.400.2278
Application: SAS Web appserver on Linux as well.

CLASSPATH="$CATALINA_BASE/lib/log4j.jar:$CATALINA_BASE/lib:$CATALINA_BASE/conf:$JRE_HOME/../lib/tools.jar:/apps/Siteminder/sdk/java/smagentapi.jar:/apps/Siteminder/sdk/java/smcrypto.jar:/apps/Siteminder/sdk/java/bc-fips-1.0.1.jar:/apps/Siteminder/sdk/java/fipsmode.jar"

After installing and configuring the SDK agent on SAS mid-tier, SDK agent can not connect to policy server.

15:23:22.707 [main] SMTRACE: SmCluster, enable, Attempting to enable cluster id = 0
15:23:22.707 [main] SMTRACE: SmCluster, enable, Attempting to enable server index = 0
15:23:22.707 [main] SMTRACE: SmServer, enable, started
15:23:22.707 [main] SMTRACE: SmServer, createConnections, Attempt to create connections = 2
15:23:22.707 [main] SMERROR: SmServer, retryTimedoutConnections, Error retrying connection NO CONN
15:23:22.708 [main] SMTRACE: SmServer, createConnection, Currently have 0 connections to server.
15:23:22.708 [main] SMINFO: SmServer, createConnection, Creating a server connection, index = 0
15:23:22.708 [main] SMTRACE: SmServerConnection, init, Attempt to init connection
15:23:23.701 [main] SMTRACE: SmAgentTcpTransport, newInstance, Using SmAgentTcpTransport class
15:23:23.702 [main] SMTRACE: SmAgentTliSession, setup, Initiating TLI handshake
15:23:23.702 [main] SMTRACE: SmConfigAttribute, decrypt, Attempting to decrypt input = {RC2}GPVycsczZW/hvg37UxNnmoI9afWwnXP0RcsBsf7iZ9.............gSRY8oHgYjLUxJPIXIgSKGA29fk
15:23:23.733 [main] SMERROR: SmServerConnection, handshake, Failed session setup.
com.ca.siteminder.sdk.agentapi.tli.SmAgentTliException: Shared secret invalid.
        at com.ca.siteminder.sdk.agentapi.tli.n.br(smagentapi_obfsc:229)
        at com.ca.siteminder.sdk.agentapi.connection.i.aQ(smagentapi_obfsc:328)
        at com.ca.siteminder.sdk.agentapi.connection.h.aH(smagentapi_obfsc:409)
        at com.ca.siteminder.sdk.agentapi.connection.h.h(smagentapi_obfsc:304)
        at com.ca.siteminder.sdk.agentapi.connection.h.an(smagentapi_obfsc:235)
        at com.ca.siteminder.sdk.agentapi.connection.c.an(smagentapi_obfsc:646)

Cause

Handshake error is just a symptom.

The underlining error is this:

catalina.out
----------------------
Caused by: com.ca.sso.smcrypto.SmCryptoLibException: org.bouncycastle.crypto.InvalidCipherTextException: Error finalising cipher data: pad block corrupted
 at com.ca.sso.smcrypto.bcfipsimpl.SmBaseCrypto.decryptBytes(SmBaseCrypto.java:421)
 at com.ca.siteminder.sdk.agentapi.config.a.t(smagentapi_obfsc:342)
 ... 29 more
Caused by: org.bouncycastle.crypto.InvalidCipherTextException: Error finalising cipher data: pad block corrupted
 at org.bouncycastle.crypto.internal.io.CipherOutputStreamImpl.close(Unknown Source)
 at com.ca.sso.smcrypto.bcfipsimpl.SmBaseCrypto.decryptBytes(SmBaseCrypto.java:384)
 ... 30 more
Caused by: org.bouncycastle.crypto.internal.InvalidCipherTextException: pad block corrupted
 at org.bouncycastle.crypto.internal.paddings.PKCS7Padding.padCount(Unknown Source)
 at org.bouncycastle.crypto.internal.paddings.PaddedBufferedBlockCipher.doFinal(Unknown Source)
 ... 32 more"

The latest Policy Server 12.8 runs Bouncy Castle Java FIPS 1.0.1, which is a dependency change from earlier 12.7 or 12.52 release.

Any newer java SDK agent must be compiled and run with newer set of SDK jars.

Environment

Release : 12.8

Component : SITEMINDER -SDK

Resolution

There are two problems.

1. CLASSPATH was misconfigured, missing or not loading the correct set of 12.8sp4 lib jars.

classpath should include these from SDK location :../../java/smagentapi.jar:../../java/smjavasdk2.jar:../../java/smcrypto.jar:../../java/bc-fips-1.0.1.jar, and remove legacy jar like fipsmode.jar

2. Customer has regular agent and SDK agent on the same server. They registered SDK agent with regular agent command and regular agent lib path, which causes the error.

Regular agent is unlike SDK agent, it uses C based code, hence regular agent is not impacted with Java jar change. SmHost.conf file created with regular agent can NOT be used with SDK agent.

When using SDK agent, one must follow these steps in the doc to do SmRegHost.

Windows platform:

SM_SMREGHOST_CLASSPATH="c:\ca\sdk\java\smagentapi.jar;c:\ca\sdk\java\smcrypto.jar;c:\ca\sdk\java\bc-fips-1.0.1.jar"
java -Dcom.ca.siteminder.sdk.agentapi.enableDebug="true" -classpath %SM_SMREGHOST_CLASSPATH% com.ca.siteminder.sdk.agentapi.SmRegHost -i 127.0.0.1 -hc host_conf1 -hn trustedhost3 -u siteminder -p password -f  "c:\ca\sdk\SmHost.conf"

Please alter above command to Linux platform before direct usage.

Linux platform:

SM_SMREGHOST_CLASSPATH=/opt/CA/sdk/java/smagentapi.jar:/opt/CA/sdk/java/smcrypto.jar:/opt/CA/sdk/java/bc-fips-1.0.1.jar

If policy server is in COMPAT mode.
java -Dcom.ca.siteminder.sdk.agentapi.enableDebug="true" -classpath $SM_SMREGHOST_CLASSPATH com.ca.siteminder.sdk.agentapi.SmRegHost -i 127.0.0.1 -hc hco -hn trustedhost3 -u siteminder -p password

If policy server is in FIPS ONLY mode.
java -Dcom.ca.siteminder.sdk.agentapi.enableDebug="true" -classpath $SM_SMREGHOST_CLASSPATH com.ca.siteminder.sdk.agentapi.SmRegHost -i 127.0.0.1 -hc hco -hn trustedhost3 -u siteminder -p password -cf ONLY

-cf option: <Crypto FIPS140 mode (COMPAT or MIGRATE or ONLY)>

After generated new SmHost.conf and corrected CLASSPATH, SDK agent is able to connect.

Additional Information

https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/programming/sdks/programming-in-java/agent-api-in-java.html

Powered by Wolken Software