[SiteMinder] Auto Start of Policy Server not working on RHEL 7.x

Products

SITEMINDER

Issue/Introduction

Auto Startup Script has been configured according to the documentation.

https://techdocs.broadcom.com/us/en/symantec-security-software/identity-security/siteminder/12-8/installing/install-a-policy-server/install-policy-server-on-unix/configure-auto-startup.html

When manually starting up policy server via command-line (start-all) the policy server starts up just fine.

But when the machine is rebooted, policy server starts but is then killed so policy server need to be started up manually.

"/var/log/messages" reports the following.

Feb 11 15:38:31 <host> systemd: Starting S98sm Service...
Feb 11 15:40:01 <host> systemd: S98sm.service start operation timed out. Terminating.
Feb 11 15:40:01 <host> systemd: Failed to start S98sm Service.
Feb 11 15:40:01 <host> systemd: Unit S98sm.service entered failed state.
Feb 11 15:40:01 <host> systemd: S98sm.service failed.

"systemctl status S98sm" reports the following.

[root@<host> log]# systemctl status S98sm
● S98sm.service - S98sm Service
   Loaded: loaded (/etc/systemd/system/S98sm.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Thu 2021-02-11 15:40:01 AEDT; 13s ago
  Process: 818 ExecStart=/{home_policy_server}/start-all (code=killed, status=TERM)

Feb 11 15:38:31 <host>.example.com systemd[1]: Starting S98sm Service...
Feb 11 02:38:32 <host>.example.com smpolicysrv[917]: EventID: 1002, Category: Authentication, Description: The server is initializing.
Feb 11 02:38:32 <host>.example.com smpolicysrv[917]: EventID: 5000, Category: Authentication, Description: The server failed to initialize.
Feb 11 15:40:01 <host>.example.com systemd[1]: S98sm.service start operation timed out. Terminating.
Feb 11 15:40:01 <host>.example.com systemd[1]: Failed to start S98sm Service.
Feb 11 15:40:01 <host>.example.com systemd[1]: Unit S98sm.service entered failed state.
Feb 11 15:40:01 <host>.example.com systemd[1]: S98sm.service failed.

Environment

Release : 12.8.x

Component : SITEMINDER POLICY SERVER

Cause

This is because the default Startup Timeout is 90 seconds.

If the service does not start within 90 seconds to return a success result then it will be killed and considered failed start.

Resolution

You can increase the StartupTimeout in the "/etc/systemd/system/S98sm.service"

Following sample shows the TimeoutStartSec being set to 3000 seconds. You may need to test to come up with sufficient value.
"infinity" is also a valid option but should rather investigate why the startup would take so long.

[Unit]
Description=S98sm Service
Requires=network.target

[Service]
Type=forking
User=smuser
ExecStart=-/{home_policy_server}/start-all
ExecStop=-/{home_policy_server}/stop-all
Restart=on-abort
TimeoutStartSec=3000

[Install]
WantedBy=multi-user.target

If the TimeoutStartSec value is sufficient, then the following should be observed.

"/var/log/messages" reports the following.

Feb 11 15:42:02 <host> systemd: Starting S98sm Service...
Feb 11 15:43:38 <host> systemd: Started S98sm Service.

"systemctl status S98sm" reports the following.

[root@smps-02 log]# systemctl status S98sm
● S98sm.service - S98sm Service
   Loaded: loaded (/etc/systemd/system/S98sm.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-02-11 15:43:38 AEDT; 23min ago
  Process: 3361 ExecStart=/{home_policy_server}/start-all (code=exited, status=1/FAILURE)
 Main PID: 3373 (smexec)
   CGroup: /system.slice/S98sm.service
           ├─3373 /{home_policy_server}/bin/smexec
           ├─3374 /{home_jvm}/jre/bin/java -Xrs -Xmx64m -Dnete.ps.root=/{home_policy_server} -classpath /{home_policy_server}/lib/smconapi.jar:/{home_policy_server}/lib/smmon.jar com.netegrity.sm...
           └─3375 smpolicysrv

Feb 11 15:42:02 <host>.example.com systemd[1]: Starting S98sm Service...
Feb 11 15:42:03 <host>.example.com smpolicysrv[3375]: EventID: 1002, Category: Authentication, Description: The server is initializing.
Feb 11 15:43:27 <host>.example.com smpolicysrv[3375]: EventID: 1000, Category: Authentication, Description: The server is up.
Feb 11 15:43:38 <host>.example.com start-all[3361]: SiteMinder Policy Server is running
Feb 11 15:43:38 <host>.example.com start-all[3361]: SiteMinder Health Monitor is running
Feb 11 15:43:38 <host>.example.com systemd[1]: Started S98sm Service.

Additional Information

More detail on TimeoutStartSec: https://bugzilla.redhat.com/show_bug.cgi?id=1446015