search cancel

Handshake error: Bad hostname in hello message


Article ID: 75120


Updated On:


CA Single Sign On Secure Proxy Server (SiteMinder) CA Single Sign On SOA Security Manager (SiteMinder) CA Single Sign-On


We have policy servers psA and psB as well as web agents agentA and agentB. Currently, they are supposed to talk one to one; so agentA to psA and agentB to psB. Therefore, the smhost registration was done from agentA to psA and agentB to psB.

Both agents seem to be up and running and the LLAWP processes are up. But for some reason, if we tail the smps.log in policy server psB, we continuously see the following error message:

[20340/4075703152][Fri Jan 19 2018 15:17:59][CServer.cpp:2006][ERROR][sm-Tunnel-00010] Bad security handshake attempt. Handshake error: 3160 [20340/4075703152]
[Fri Jan 19 2018 15:17:59][CServer.cpp:2027][ERROR][sm-Tunnel-00100] Handshake error: Bad hostname in hello message [20340/4075703152]
[Fri Jan 19 2018 15:17:59][CServer.cpp:2178][ERROR][sm-Server-01070] Failed handshake with [20340/4117662576][Fri Jan 19 2018 15:18:29][CServer.cpp:1912][ERROR][sm-Server-01060] Handshake error: Unknown client name '' in hello message

The IP "" and host name '' are that of agentA.

So, it seems agentA is trying to communicate or send a "hello" message to policy psB. For troubleshooting, we deleted the TrustedHost of agentA and agentB in each policy servers through XPSExplorer. We also deleted the existing SmHost.conf files in each agent machines, then re-registered the hosts one to one (i.e. agentA to psA and agentB to psB). However, the same error message appears. We suspected  there is data stored somewhere on agentA's machine telling it to communicate with psB, but where would this be? Where can we clear this information? We assumed that the SmHost.conf was the only place that stored such information.


Possible cause we suspect is that the replication was not working correctly at some point earlier, causing the deleted TrustedHost of agentA in the psB policy store to remain in the replicated store of psB.


Policy Servers and Agents at version R12.52SP1CR04.
OS: RH Linux 6.9


Found the unwanted TrustedHost that remained in the replicated store using XPSExplorer, and deleted.