Handshake error when using SM Test Tool from a different box


Article ID: 16721


Updated On:


CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign-On


We have installed SDK and we are trying to use the SM Test Tool included to do some tests for our custom authentication scheme from the same box. However after the host registration was done correctly (using smreghost and generating a SmHost.conf) we are getting the following error when we try to connect to the Policy Server, using the SmHost.conf from registration:

"Cannot obtain host configuration information using specified SmHost.conf file"

In the smps.log we see the following:

[25/12/2017][14:29:55][1234][1092][CServer.cpp:2058][CAgentMessageHandler::DoWork][][63940][New connection attempt from client host]
[25/12/2017][14:29:55][1234][1092][CServer.cpp:1842][GetSecretFunc][Getting current secret for the Agent testtoolhost]
[25/12/2017][14:29:55][1234][1092][CServer.cpp:1899][GetSecretFunc][Getting previous secret for the Agent testtoolhost]
[25/12/2017][14:29:55][1234][1092][CServer.cpp:1905][GetSecretFunc][Error while fetching previous secret for the Agent testtoolhost]
[25/12/2017][14:29:55][1234][1092][CServer.cpp:1948][LogMessage:ERROR: Bad security handshake attempt. Handshake error: 3154]
[25/12/2017][14:29:55][1234][1092][CServer.cpp:1959][LogMessage:ERROR: Handshake error: Shared secret incorrect for this client]
[25/12/2017][14:29:55][1234][1092][CServer.cpp:2121][LogMessage:ERROR: Failed handshake with]
[25/12/2017][14:29:55][1234][1092][CServer.cpp:2127][CAgentMessageHandler::DoWork][][63940][Handshake error with trusted host testtoolhost with IP on Port No 63940]
[25/12/2017][14:29:55][1234][1092][CServer.cpp:3054][CAgentMessageHandler::HandleClose][][63940][Ending client session #215758]

If we test from the Policy Server SM Test Tool using the same SmHost.conf, it works.

How can we make it work in the box where the SDK is installed?


SDK R12.52 SP1


This error happens because the SDK installer does not install automatically any CAPKI libraries which are needed when you have to use the SM Test Tool. It works in the Policy Server, as this one installs the CAPKI libraries, so this will work in any machine where you have installed a component which installs the CAPKI libraries, like the Web Agent or Access Gateway(SPS). You can check this at the following location:

R12.52 SP1 - Known Issues - Installation of ETPKI Libraries

In order to solve this, you need to install the CAPKI (formerly ETPKI) libraries manually by using the installer included in the SDK path:

<SDK_install_path>/etpki-install   (for 32 bit)
<SDK_install_path>/etpki-install-64    (for 64 bit)

Here, you need to run the setup tool included as follows:

setup install caller=smtesttool instdir=<install_path>

Where install_path will be the installation directory for the CAPKI libraries. Optionally, you can specify the verbose parameter so you will have some output files to warn you when the process is finishes, as the setup tool is a silent installer.

For Linux, optionally you can specify the following parameter to enable setting environment variables for the specified users:


none: do not set environment variables (default; it may require root privileges depending on the installation directory)
user: current user only ($HOME/.profile)
all : all users (for using this, login must be root).
- Note: If /etc/profile should not be updated as part of CAPKI installation (with env=all option), then Update_Profile=0 should be set in the environment before the installation of CAPKI.


After the command is triggered, you will see a tmp folder created in the destination path that will appear until the installation finishes. You can find a log file in the Windows temp folder (%TEMP%/capki_install.log) or for Linux in the /tmp folder (/tmp/capki_install.log). When finished, for Windows you will need to restart the machine so the new registry entries set are loaded.