After upgrading the NetOps Portal from version 3.7.14 or earlier, or if upgrading from 20.2.2 or earlier, to version 21.2.1, the Event Manager data source will not synchronize.
The Data Sources page also shows the Event Manager data source on the same version as prior to the upgrade.
In the (default path) /opt/CA/PerformanceCenter/EM/logs/EMService.log we see this error:
ERROR | qtp66491224-198 | YYYY-MM-DD HH:MM:SS,XXX | com.ca.im.portal.api.security.Encryption
| Error performing encryption operation
javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted
at org.bouncycastle.jcajce.provider.BaseCipher.engineDoFinal(Unknown Source)
at javax.crypto.Cipher.doFinal(Cipher.java:2168)
at com.ca.im.portal.api.security.Encryption.doOperation(Encryption.java:207)
at com.ca.im.portal.api.security.Encryption.decryptFromBytes(Encryption.java:138)
at com.ca.im.portal.api.security.Encryption.decrypt(Encryption.java:127)
at com.ca.im.portal.api.security.Encrypter.decrypt(Encrypter.java:73)
at com.ca.im.portal.api.security.SsoToken.parseToken(SsoToken.java:91)
at com.ca.im.portal.common.web.util.GlobalAdminAuthInterceptor.validateUsingSsoToken(GlobalAdminAuthInterceptor.java:199)
at com.ca.im.portal.common.web.util.GlobalAdminAuthInterceptor.handleMessage(GlobalAdminAuthInterceptor.java:95)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:228)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
at java.lang.Thread.run(Thread.java:748)
WARN | qtp66491224-198 | YYYY-MM-DD HH:MM:SS,### | com.ca.im.portal.common.web.util.GlobalAdminAuthInterceptor
| SsoToken is expired or could not be decrypted
In both the (default paths) /opt/CA/PerformanceCenter/PC/logs/PCService.log and the /opt/CA/PerformanceCenter/DM/logs/DMService.log we see this error:
ERROR | qtp341763619-246 | 2021-11-03 13:16:14,316 | com.ca.im.portal.api.services.datasource.DataSourcePoll
| Received WebServiceException from version check for data source Event Manager@<EM_Host>. CAUSE=org.apache.cxf.transport.http.HTTPException: HTTP response '401: Unauthorized' when communicating with http://<EM_Host>:8281/EventManager/DataSourceWS.asmx. MESSAGE=Could not send Message.. Returning DS_COMM_FAILURE result.
ERROR | qtp341763619-246 | 2021-11-03 13:16:14,316 | com.ca.im.portal.api.services.datasource.DataSourcePoll
| javax.xml.ws.WebServiceException: Could not send Message.
All supported DX NetOps Performance Management releases r21.2.1 and earlier that utilize the SsoEncryptionDecryptionKey values.
The upgrade fails to copy the SsoEncryptionDecryptionKey value to the Event Manager MySql em database.
This is addressed via defect DE509973 which is resolved starting with the r21.2.2 NetOps release.
To resolve this issue in r21.2.1 without upgrading to a newer current release follow these steps.
NOTE: You will need the Netops Portal MySQL password to enter when prompted.
Confirm the errors are gone from the logs and the Event Manager Data Source is again successfully syncing in the Portal web UI.
This is a sample of a working system. Note the highest Priority value for the netqosportal DB (0) has a value that matches the highest Priority value (1) from the em DB. If these did not match we'd update the em DB Priority value 1 to match the netqosportal DB Priority value 0 value.
This can also happen if a local override is set for "SsoEncryptionDecryptionKey". In that scenario you will see 2 values for each query.
In that scenario where normally netqosportal only has a value set for Priority 0 we'd see it set with a Priority 1 value as well.
If that is found use the (default path) /opt/CA/PerformanceCenter/SsoConfig tool to reset the value for the SingleSign-On SsoEncryptionDecriptionKey value.