Many times when an agent fails to connect to a policy server, is due to FIPS mode from both sides are not the same.
Often it exhibits as handshake error on policy server side.
How to identify if SiteMinder is on FIPS only mode or not?
Release : 12.8
Component : SITEMINDER -SDK
FIPS only mode is part of configuration choice during SiteMinder installation.
Agent and policy server needs to be on the same mode in order for the sharedsecret to be correctly decrypted.
From SmHost.conf content, both fields indicate this agent is running on FIPS only mode. If this is not FIPS only mode, then sharedsecret will be RC2 encryption instead,
and fipsmode value will be something else.
sharedsecret="{AES}lNxBlQfbIag............"
...
...
fipsmode="ONLY"
Also note that in the WebAgent.log, upon startup, after it reporting the Agent version, it will note the FIPS mode.
Example of non FIPS mode
[1986232/2483025664][Mon Feb 24 2025 20:20:29] FIPS 140 Cryptographic Mode is 'non-FIPS (compatibility)'.
smps.log:
[3436/3440][Sun Sep 19 2021 15:51:40][CServer.cpp:4193][INFO][sm-Server-04450] Policy Server employing only FIPS-140 cryptographic algorithms.
https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/upgrading/migrate-your-environment-to-use-fips-compliant-algorithms/configure-fips-only-mode.html