Scheduled Report Repository Issue
Article ID: 273251
Updated On:
Products
Issue/Introduction
We are getting a system status notice in Portal:
Scheduled Report Repository
Operational Status =Degraded
We have 100g file system for /opt
Below are the top files consuming more space.
2490380 340468 -rw-r--r-- 1 adminUser adminUser 348618768 Sep 1 23:59 /opt/CA/PerformanceCenter/EM/logs/wrapper-20230901.log
2363010 524292 -rw-r----- 1 mysql mysql 536870912 Sep 5 07:43 /opt/CA/MySql/data/em/events_l10n#p#pmax.ibd
2363012 745476 -rw-r----- 1 mysql mysql 763363328 Sep 5 10:46 /opt/CA/MySql/data/em/event_properties#p#pmax.ibd
2490383 8076956 -rw-r--r-- 1 adminUser adminUser 8270796318 Sep 5 10:58 /opt/CA/PerformanceCenter/EM/logs/wrapper-20230905.log
2490372 13765548 -rw-r--r-- 1 adminUser adminUser 14095911308 Sep 3 00:00 /opt/CA/PerformanceCenter/EM/logs/wrapper-20230902.log
2490377 18102856 -rw-r--r-- 1 adminUser adminUser 18537320224 Sep 5 00:00 /opt/CA/PerformanceCenter/EM/logs/wrapper-20230904.log
2490386 18544808 -rw-r--r-- 1 adminUser adminUser 18989877566 Sep 4 00:00 /opt/CA/PerformanceCenter/EM/logs/wrapper-20230903.log
NOTE : we are also seeing no events in Portal.
EM wrapper logs are rapidly filling the hard drive.
Environment
Release : 22.2
Cause
The DA and Portal systems were out of time synch.
We see these messages in the logs:
/opt/CA/PerformanceCenter/EM/logs/wrapper-20230901.log
INFO | jvm 1 | 2023/09/06 09:37:07 | --------------------------------------
INFO | jvm 1 | 2023/09/06 09:37:07 | WARN | qtp500999331-156946 | 2023-09-06 09:37:07,853 | com.ca.im.portal.common.web.util.AuthInterceptor
INFO | jvm 1 | 2023/09/06 09:37:07 | | SsoToken is expired in request http://1.1.1.1:8281/EventManager/EventManagerWS.asmx
INFO | jvm 1 | 2023/09/06 09:37:07 | Sep 06, 2023 9:37:07 AM org.apache.cxf.services.EventManagerWS.EventManagerWSSoap.EventManagerWSSoap
INFO | jvm 1 | 2023/09/06 09:37:07 | INFO: Inbound Message
INFO | jvm 1 | 2023/09/06 09:37:07 | ----------------------------
INFO | jvm 1 | 2023/09/06 09:37:07 | ID: 1766235
INFO | jvm 1 | 2023/09/06 09:37:07 | Address: http://1.1.1.1:8281/EventManager/EventManagerWS.asmx
INFO | jvm 1 | 2023/09/06 09:37:07 | Encoding: UTF-8
INFO | jvm 1 | 2023/09/06 09:37:07 | Http-Method: POST
INFO | jvm 1 | 2023/09/06 09:37:07 | Content-Type: text/xml; charset=UTF-8
INFO | jvm 1 | 2023/09/06 09:37:07 | Headers: {Accept=[*/*], Cache-Control=[no-cache], connection=[keep-alive], content-type=[text/xml; charset=UTF-8], Cookie=[CADefaultCookie=w9wFppK5sx0
********************************************
/opt/IMDataAggregator/apache-karaf/data/log/karaf.log
WARN | EventPush | 2023-09-07T16:09:50,642 | PhaseInterceptorChain | ache.cxf.common.logging.LogUtils 465 | org.apache.cxf.cxf-core | | Interceptor for {http://netqos.com/nqevents/EventManager}IEventManagerWSService#{http://netqos.com/nqevents/EventManager}Push has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message.
at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:67) ~[bundleFile:3.5.5]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307) ~[bundleFile:3.5.5]
at org.apache.cxf.endpoint.ClientImpl.doInvoke(ClientImpl.java:528) ~[bundleFile:3.5.5]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:439) ~[bundleFile:3.5.5]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:354) ~[bundleFile:3.5.5]
at org.apache.cxf.endpoint.ClientImpl.invoke(ClientImpl.java:312) ~[bundleFile:3.5.5]
at org.apache.cxf.frontend.ClientProxy.invokeSync(ClientProxy.java:96) ~[bundleFile:3.5.5]
at org.apache.cxf.jaxws.JaxWsClientProxy.invoke(JaxWsClientProxy.java:140) ~[bundleFile:3.5.5]
at com.sun.proxy.$Proxy450.push(Unknown Source) ~[?:?]
at com.ca.im.connector.eventproducer.EventProducerWS.pushEvents(EventProducerWS.java:989) ~[bundleFile:?]
at com.ca.im.connector.eventproducer.EventProducerWS$1.run(EventProducerWS.java:429) ~[bundleFile:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: org.apache.cxf.transport.http.HTTPException: HTTP response '401: Unauthorized' when communicating with http://1.1.1.1:8281/EventManager/EventManagerWS.asmx
******************************
Resolution
Fix the time synch between all NetOps hosts.
Once fixed, the tokens will match again.
Additional Information
Feedback
