SmAgentTliException: "Shared secret invalid" Error When Starting a SiteMinder SDK Agent

Products

SITEMINDER CA Single Sign On Agents (SiteMinder)

Issue/Introduction

After installing and configuring an SDK Agent on the SaS mid-tier, the SDK Agent cannot connect to the Policy Server.

15:23:22.707 [main] SMTRACE: SmCluster, enable, Attempting to enable cluster id = 0
15:23:22.707 [main] SMTRACE: SmCluster, enable, Attempting to enable server index = 0
15:23:22.707 [main] SMTRACE: SmServer, enable, started
15:23:22.707 [main] SMTRACE: SmServer, createConnections, Attempt to create connections = 2
15:23:22.707 [main] SMERROR: SmServer, retryTimedoutConnections, Error retrying connection NO CONN
15:23:22.708 [main] SMTRACE: SmServer, createConnection, Currently have 0 connections to server.
15:23:22.708 [main] SMINFO: SmServer, createConnection, Creating a server connection, index = 0
15:23:22.708 [main] SMTRACE: SmServerConnection, init, Attempt to init connection
15:23:23.701 [main] SMTRACE: SmAgentTcpTransport, newInstance, Using SmAgentTcpTransport class
15:23:23.702 [main] SMTRACE: SmAgentTliSession, setup, Initiating TLI handshake
15:23:23.702 [main] SMTRACE: SmConfigAttribute, decrypt, Attempting to decrypt input = {RC2}GPVycs...................................................................A29fk
15:23:23.733 [main] SMERROR: SmServerConnection, handshake, Failed session setup.
com.ca.siteminder.sdk.agentapi.tli.SmAgentTliException: Shared secret invalid.
at com.ca.siteminder.sdk.agentapi.tli.n.br(smagentapi_obfsc:229)
at com.ca.siteminder.sdk.agentapi.connection.i.aQ(smagentapi_obfsc:328)
at com.ca.siteminder.sdk.agentapi.connection.h.aH(smagentapi_obfsc:409)
at com.ca.siteminder.sdk.agentapi.connection.h.h(smagentapi_obfsc:304)
at com.ca.siteminder.sdk.agentapi.connection.h.an(smagentapi_obfsc:235)
at com.ca.siteminder.sdk.agentapi.connection.c.an(smagentapi_obfsc:646)

The JVM CLASSPATH is configured as follows:

CLASSPATH="$CATALINA_BASE/lib/log4j.jar:$CATALINA_BASE/lib:$CATALINA_BASE/conf:$JRE_HOME/../lib/tools.jar:/{home_sdk}/java/smagentapi.jar:/{home_sdk}/java/smcrypto.jar:/{home_sdk}/java/bc-fips-1.*.*.jar:/{home_sdk}/java/fipsmode.jar"

Environment

SDK 12.8
OS: Red Hat Enterprise Linux Server release 7.8
Policy Server Version: 12.80.400.2278
Application: SaS web application server, also running on Linux.

Cause

The handshake error is a symptom of the underlying problem.

The underlying error is shown below — decryption fails during the encryption handling:

catalina.out:

Caused by: com.ca.sso.smcrypto.SmCryptoLibException: org.bouncycastle.crypto.InvalidCipherTextException: Error finalising cipher data: pad block corrupted
at com.ca.sso.smcrypto.bcfipsimpl.SmBaseCrypto.decryptBytes(SmBaseCrypto.java:421)
at com.ca.siteminder.sdk.agentapi.config.a.t(smagentapi_obfsc:342)
...

Caused by: org.bouncycastle.crypto.InvalidCipherTextException: Error finalising cipher data: pad block corrupted
at org.bouncycastle.crypto.internal.io.CipherOutputStreamImpl.close(Unknown Source)
at com.ca.sso.smcrypto.bcfipsimpl.SmBaseCrypto.decryptBytes(SmBaseCrypto.java:384)
...

Caused by: org.bouncycastle.crypto.internal.InvalidCipherTextException: pad block corrupted
at org.bouncycastle.crypto.internal.paddings.PKCS7Padding.padCount(Unknown Source)
at org.bouncycastle.crypto.internal.paddings.PaddedBufferedBlockCipher.doFinal(Unknown Source)
...

The latest Policy Server 12.8 runs Bouncy Castle Java FIPS 1.*.*, which is a dependency change from the earlier 12.7 and 12.52 releases.

Any newer Java SDK Agent must be compiled and run with a newer set of SDK jars.

Resolution

There are two problems:

The CLASSPATH is misconfigured — it is missing, or not loading, the correct set of 12.8 library jars.

The CLASSPATH should include the following jars from the SDK location: .../java/smagentapi.jar:.../java/smjavasdk2.jar:.../java/smcrypto.jar:.../java/bc-fips-1.*.*.jar, and legacy jars such as fipsmode.jar should be removed.
A Web Agent and an SDK Agent are running on the same server. The SDK Agent has been registered using the Web Agent's registration command and library PATH, which causes the error.

The Web Agent is unlike the SDK Agent — it uses C-based code, so it is not impacted by the Java jar change. The SmHost.conf file created for the Web Agent cannot be used with the SDK Agent.

When using an SDK Agent, follow the SmRegHost for the Pure Java steps (1).

Windows:

c:\> set SM_SMREGHOST_CLASSPATH="c:\ca\sdk\java\smagentapi.jar;c:\ca\sdk\java\smcrypto.jar;c:\ca\sdk\java\bc-fips-1.*.*.jar"
c:\> java -Dcom.ca.siteminder.sdk.agentapi.enableDebug="true" -classpath %SM_SMREGHOST_CLASSPATH% com.ca.siteminder.sdk.agentapi.SmRegHost -i <Policy-Server-IP> -hc <hc> -hn <hn> -u <super_user> -p <superuser_password> -f "C:\{home_policy_server}\SmHost.conf"
Linux:

# export SM_SMREGHOST_CLASSPATH=/{home_sdk}/java/smagentapi.jar:/{home_sdk}/java/smcrypto.jar:/{home_sdk}/java/bc-fips-1.*.*.jar

If the Policy Server is in COMPAT mode, the command will be:

# java -Dcom.ca.siteminder.sdk.agentapi.enableDebug="true" -classpath $SM_SMREGHOST_CLASSPATH com.ca.siteminder.sdk.agentapi.SmRegHost -i <Policy-Server-IP> -hc <hc> -hn <hn> -u <super_user> -p <superuser_password>

If the Policy Server is in FIPS ONLY mode, the command is slightly different — add the parameter -cf ONLY:

# java -Dcom.ca.siteminder.sdk.agentapi.enableDebug="true" -classpath $SM_SMREGHOST_CLASSPATH com.ca.siteminder.sdk.agentapi.SmRegHost -i <Policy-Server-IP> -hc <hc> -hn <hn> -u <super_user> -p <superuser_password> -cf ONLY

Note: The -cf option can have the following values:

COMPAT
MIGRATE
ONLY

After generating the new SmHost.conf file and correcting the CLASSPATH, the SDK Agent is able to connect.

Additional Information

Agent API in Java