Event Manager data source will not sync after Netops Portal upgrade to version 21.2.1

book

Article ID: 220330

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration

Issue/Introduction

After upgrading the NetOps Portal from version 3.7.14 or earlier, or if upgrading from 20.2.2 or earlier, to version 21.2.1, the Event Manager data source will not synchronize.

The Data Sources page also shows the Event Manager data source on the same version as prior to the upgrade.

In the /opt/CA/PerformanceCenter/EM/logs/EMService.log we see the error:

ERROR | qtp66491224-198          | YYYY-MM-DD HH:MM:SS,XXX | com.ca.im.portal.api.security.Encryption                         
      | Error performing encryption operation
javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted
 at org.bouncycastle.jcajce.provider.BaseCipher.engineDoFinal(Unknown Source)
 at javax.crypto.Cipher.doFinal(Cipher.java:2168)
 at com.ca.im.portal.api.security.Encryption.doOperation(Encryption.java:207)
 at com.ca.im.portal.api.security.Encryption.decryptFromBytes(Encryption.java:138)
 at com.ca.im.portal.api.security.Encryption.decrypt(Encryption.java:127)
 at com.ca.im.portal.api.security.Encrypter.decrypt(Encrypter.java:73)
 at com.ca.im.portal.api.security.SsoToken.parseToken(SsoToken.java:91)
 at com.ca.im.portal.common.web.util.GlobalAdminAuthInterceptor.validateUsingSsoToken(GlobalAdminAuthInterceptor.java:199)
 at com.ca.im.portal.common.web.util.GlobalAdminAuthInterceptor.handleMessage(GlobalAdminAuthInterceptor.java:95)
 at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
 at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
 at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)
 at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
 at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
 at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
 at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225)
 at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298)
 at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
 at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273)
 at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
 at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
 at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:228)
 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
 at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
 at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
 at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
 at org.eclipse.jetty.server.Server.handle(Server.java:516)
 at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
 at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
 at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
 at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
 at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
 at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
 at java.lang.Thread.run(Thread.java:748)
WARN  | qtp66491224-198          | YYYY-MM-DD HH:MM:SS,### | com.ca.im.portal.common.web.util.GlobalAdminAuthInterceptor    
      | SsoToken is expired or could not be decrypted

 

And in the /opt/CA/PerformanceCenter/EM/logs/DMService.log we see the error:

INFO  | jvm 1   | YYYY-MM-DD HH:MM:SS |      | Received WebServiceException from version check for data source Event [email protected] Address. CAUSE=org.apache.cxf.transport.http.HTTPException: HTTP response '401: Unauthorized' when communicating with http://IP Address:8281/EventManager/DataSourceWS.asmx. MESSAGE=Could not send Message.. Returning c result.

com.ca.im.portal.api.services.datasource.DataSourceBuilderImpl  
INFO  | jvm 1   | YYYY-MM-DD HH:MM:SS |      | commit failed - exception: enum.datasourceerror.DS_COMM_FAILURE

 

Cause

Defect: DE509973

Environment

Release : 21.2

Component : PERFORMANCE MANAGEMENT INSTALLATIONS/UPGRADES

Resolution

Defect DE509973 is currently under investigation and this article will be updated as this investigation progresses. 

WORKAROUND POST-UPGRADE:

NOTE: You will need the Netops Portal MySQL password to enter when prompted.
 

1) Verify that the SsoEncryptionDecryptionKey does not match between the netqosportal and em databases.
   The em database will have the old non-unique SsoEncryptionDecryptionKey.

   The netqosportal value will be needed in a later step:

a) Get the value of SsoEncryptionDecryptionKey from the netqosportal database:

mysql -u netqos -p -e "select PropValue from netqosportal.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey'"
Enter password:
+-----------+
| PropValue |
+-----------+
| btILZiyG  |
+-----------+

Note that the value will be different in your installation. 

b) Get the value of SsoEncryptionDecryptionKey from the em database:

mysql -u netqos -p -e "select PropValue from em.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey' and Priority = 1"
Enter password:
+-----------+
| PropValue |
+-----------+
| #$utP9%z  |
+-----------+

 

2) Stop the Event Manager service:

systemctl stop caperfcenter_eventmanager


3) Update the em database to match the netqosportal database:

mysql -u netqos -p -e "replace into em.performance_center_properties values ('SsoEncryptionDecryptionKey',1,'btILZiyG','N',UNIX_TIMESTAMP());"

In the example above btILZiyG woudl be replaced with the value from step 1a.

4) Verify the value is updated:

mysql -u netqos -p -e "select PropValue from em.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey' and Priority = 1"
Enter password:
+-----------+
| PropValue |
+-----------+
| btILZiyG  |
+-----------+

5) Start the Event Manager service:

systemctl start caperfcenter_eventmanager

Additional Information

This can also happen if a local override is set for "SsoEncryptionDecryptionKey" and then you may see 2 values for each query 

PM

|  FW5kc3U2 |
| FWkc3U1 

EM
 | #OT5eU3  |
| FW5kc3Ul  |

Go into /opt/CA/PerformanceCenter/SsoConfig and reset the value for SSO - SsoEncryptionDecriptionKey