Many times when an agent fails to connect to a policy server, is due to FIPS mode from both sides are not the same.
Often it exhibits as handshake error on policy server side.
How to identify if SiteMinder is on FIPS only mode or not?
Release : 12.8
Component : SITEMINDER -SDK
FIPS only mode is part of configuration choice during SiteMinder installation.
Agent and policy server needs to be on the same mode in order for the sharedsecret to be correctly decrypted.
From SmHost.conf content, both fields indicate this agent is running on FIPS only mode. If this is not FIPS only mode, then sharedsecret will be RC2 encryption instead,
and fipsmode value will be something else.
sharedsecret="{AES}lNxBlQfbIag............"
...
...
fipsmode="ONLY"
smps.log:
[3436/3440][Sun Sep 19 2021 15:51:40][CServer.cpp:4193][INFO][sm-Server-04450] Policy Server employing only FIPS-140 cryptographic algorithms.
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/upgrading/migrate-your-environment-to-use-fips-compliant-algorithms/configure-fips-only-mode.html