Event Manager sync fails post Netops Portal upgrade to r21.2.1 or later
search cancel

Event Manager sync fails post Netops Portal upgrade to r21.2.1 or later

book

Article ID: 220330

calendar_today

Updated On:

Products

CA Performance Management - Usage and Administration DX NetOps

Issue/Introduction

After upgrading the NetOps Portal from version 3.7.14 or earlier, or if upgrading from 20.2.2 or earlier, to version 21.2.1, the Event Manager data source will not synchronize.
This has also been seen in later releases as well.

The Data Sources page also shows the Event Manager data source on the same version as prior to the upgrade.

In the (default path) /opt/CA/PerformanceCenter/EM/logs/EMService.log we see this error:

ERROR | qtp66491224-198          | YYYY-MM-DD HH:MM:SS,XXX | com.ca.im.portal.api.security.Encryption                         
      | Error performing encryption operation
javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted
 at org.bouncycastle.jcajce.provider.BaseCipher.engineDoFinal(Unknown Source)
 at javax.crypto.Cipher.doFinal(Cipher.java:2168)
 at com.ca.im.portal.api.security.Encryption.doOperation(Encryption.java:207)
 at com.ca.im.portal.api.security.Encryption.decryptFromBytes(Encryption.java:138)
 at com.ca.im.portal.api.security.Encryption.decrypt(Encryption.java:127)
 at com.ca.im.portal.api.security.Encrypter.decrypt(Encrypter.java:73)
 at com.ca.im.portal.api.security.SsoToken.parseToken(SsoToken.java:91)
 at com.ca.im.portal.common.web.util.GlobalAdminAuthInterceptor.validateUsingSsoToken(GlobalAdminAuthInterceptor.java:199)
 at com.ca.im.portal.common.web.util.GlobalAdminAuthInterceptor.handleMessage(GlobalAdminAuthInterceptor.java:95)
 at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
 at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
 at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:265)
 at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
 at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
 at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
 at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:225)
 at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:298)
 at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:217)
 at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
 at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:273)
 at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443)
 at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
 at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
 at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:228)
 at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
 at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
 at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
 at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
 at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
 at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
 at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
 at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
 at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
 at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
 at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
 at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
 at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
 at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
 at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
 at org.eclipse.jetty.server.Server.handle(Server.java:516)
 at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
 at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
 at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
 at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
 at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
 at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
 at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
 at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
 at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
 at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)
 at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)
 at java.lang.Thread.run(Thread.java:748)
WARN  | qtp66491224-198          | YYYY-MM-DD HH:MM:SS,### | com.ca.im.portal.common.web.util.GlobalAdminAuthInterceptor      
      | SsoToken is expired or could not be decrypted

In both the (default paths) /opt/CA/PerformanceCenter/PC/logs/PCService.log and the /opt/CA/PerformanceCenter/DM/logs/DMService.log we see this error:

ERROR | qtp341763619-246         | 2021-11-03 13:16:14,316 | com.ca.im.portal.api.services.datasource.DataSourcePoll          
      | Received WebServiceException from version check for data source Event Manager@<EM_Host>.  CAUSE=org.apache.cxf.transport.http.HTTPException: HTTP response '401: Unauthorized' when communicating with http://<EM_Host>:8281/EventManager/DataSourceWS.asmx. MESSAGE=Could not send Message..  Returning DS_COMM_FAILURE result.
ERROR | qtp341763619-246         | 2021-11-03 13:16:14,316 | com.ca.im.portal.api.services.datasource.DataSourcePoll          
      | javax.xml.ws.WebServiceException: Could not send Message.

Environment

All supported DX NetOps Performance Management releases

Cause

The upgrade fails to copy the SsoEncryptionDecryptionKey value to the Event Manager MySql em database.

Resolution

One instance of thsis was addressed via defect DE509973 which is resolved starting with the r21.2.2 NetOps release.
There have been the same issues seen in later releases.

To resolve this issue in r21.2.1 without upgrading to a newer current release follow these steps.

NOTE: You will need the Netops Portal MySQL password to enter when prompted.

  1. Gather and compare the SsoEncryptionDecryptionKey value between the netqosportal and em databases.
    1. Connect to the MySql DB using (default path) this command. Enter the password when prompted.
      1. /opt/CA/MySql/bin/mysql -uroot -p
    2. Run the following to show the value for the netqosportal DB:
      1. select PropValue,Priority from netqosportal.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
    3. Run the following to show the value for the em DB:
      1. select PropValue,Priority from em.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
  2. The values at the highest Priority value shown should match.
    1. If the values do not match for the highest Priority value shown, continue with the remaining steps below to resolve the problem.
    2. IF the values do match for the highest Priority value shown, contact support for additional assistance.
  3. Stop the Event Manager service using the command:
    1. systemctl stop caperfcenter_eventmanager
  4. In the MySql prompt run the following to update the em database SsoEncryptionDecryptionKey value to match the netqosportal database value.

    1. If you are on DX Netsop 23.3.3 or earlier run:

      replace into em.performance_center_properties values ('SsoEncryptionDecryptionKey',1,'<keyFrom_netqosportal_DB>','N',UNIX_TIMESTAMP());"

    2. If you are on 23.3.4 or later run:

      a) If the keys match but the K value is 0 run:

      update em.performance_center_properties set k=1 where propname='SsoEncryptionDecryptionKey' and priority=1;

      b) If the priority 1 key does not match the netqosportal key run:

       replace into em.performance_center_properties values ('SsoEncryptionDecryptionKey',1,'<keyFrom_netqosportal_DB>','N',UNIX_TIMESTAMP() ,1);"


  5. Run the following in the MySql prompt to verify the value is updated and matches in both databases.
    1. Run the following to show the value for the netqosportal DB:
      1. select PropValue,Priority from netqosportal.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
    2. Run the following to show the value for the em DB:
      1. select PropValue,Priority from em.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
    3. Confirm the highest Priority value from both has a matching value.
  6. Start the Event Manager service using the command:
    1. systemctl start caperfcenter_eventmanager

Confirm the errors are gone from the logs and the Event Manager Data Source is again successfully syncing in the Portal web UI.

This is a sample of a working system. Note the highest Priority value for the netqosportal DB (0) has a value that matches the highest Priority value (1) from the em DB. If these did not match we'd update the em DB Priority value 1 to match the netqosportal DB Priority value 0 value.

Additional Information

This can also happen if a local override is set for "SsoEncryptionDecryptionKey". In that scenario you will see 2 values for each query.

In that scenario where normally netqosportal only has a value set for Priority 0 we'd see it set with a Priority 1 value as well.

If that is found use the (default path) /opt/CA/PerformanceCenter/SsoConfig tool to reset the value for the SingleSign-On SsoEncryptionDecriptionKey value.

Powered by Wolken Software