SSO and event manager wont start
Article ID: 389704
Updated On: 03-03-2025
Products
Issue/Introduction
Restarted services on the portal.
SSO and EM services are failing.
For SSO, journalctl -xe shows”
…
Mar 03 08:45:20 PortalHost systemd[1]: Starting DX NetOps Portal SSO...
-- Subject: Unit caperfcenter_sso.service has begun start-up
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
--
-- Unit caperfcenter_sso.service has begun starting up.
Mar 03 08:45:20 PortalHost caperfcenter_sso[6201]: Starting DX NetOps Portal SSO...
Mar 03 08:45:20 PortalHost caperfcenter_sso[6201]: CEF:0|Broadcom|DX NetOps Portal SSO|24.3.6|100|STARTING SERVICE|1|user=mysql
Mar 03 08:45:30 PortalHost sshd[6387]: error: kex_exchange_identification: Connection closed by remote host
Mar 03 08:45:33 PortalHost sshd[6400]: error: kex_exchange_identification: Connection closed by remote host
Mar 03 08:45:40 PortalHost caperfcenter_sso[6201]: Waiting for DX NetOps Portal SSO.......................
Mar 03 08:45:40 PortalHost caperfcenter_sso[6201]: WARNING: DX NetOps Portal SSO may have failed to start.
Mar 03 08:45:40 PortalHost caperfcenter_sso[6201]: See "/opt/CA/PerformanceCenter/sso/logs/wrapper-YYYYMMDD.log" for details.
Mar 03 08:45:40 PortalHost systemd[1]: caperfcenter_sso.service: Control process exited, code=exited status=1
Mar 03 08:45:40 PortalHost systemd[1]: caperfcenter_sso.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
********************************************
In the SSO wrapper log we see this:
…….
ERROR | wrapper | 2025/03/03 08:00:47 | JVM exited while loading the application.
INFO | jvm 1 | 2025/03/03 08:00:47 | Error opening zip file or JAR manifest missing : /home/adminuser/splunk-otel-javaagent.jar
INFO | jvm 1 | 2025/03/03 08:00:47 | Error occurred during initialization of VM
INFO | jvm 1 | 2025/03/03 08:00:47 | agent library failed to init: instrument
STATUS | wrapper | 2025/03/03 08:00:52 | JVM process is gone.
Environment
DX NetOps 24.3.x
Cause
At some point a while ago, customer was experimenting with adding a splunk agent into portal.
**********
the sso wrapper-user.conf we see these entries related to a splunk agent:
……
# Splunk settings
wrapper.java.additional.30=-javaagent:/home/adminuser/splunk-otel-javaagent.jar
wrapper.java.additional.31=-Dotel.resource.attributes=service.name=sso-app,environment=NetopsPM
Resolution
Comment out the splunk entries in ALL wrapper-user.conf files (SSO, EM, DM, CONSOLE)
Restart all services.
Additional Information
PortalHost is a generic replacement, the logs will show the actual hostname.
adminuser is a generic replacement, the logs will show the correct local path.
There is an ootb splunk integration included in Portal:
Feedback
