Event Manager sync fails post Netops Portal upgrade

Products

CA Performance Management Network Observability

Issue/Introduction

The Event Manager data source will not synchronize.

The Event Manager data source shows Unable to Contact

The Data Sources page also shows the Event Manager data source on the same version as prior to the upgrade.

In the (default path) /opt/CA/PerformanceCenter/EM/logs/EMService.log we see this error:

ERROR | qtp66491224-198          | YYYY-MM-DD HH:MM:SS,XXX | com.ca.im.portal.api.security.Encryption                         
      | Error performing encryption operation
javax.crypto.BadPaddingException: Error finalising cipher data: pad block corrupted
 at org.bouncycastle.jcajce.provider.BaseCipher.engineDoFinal(Unknown Source)

WARN  | qtp66491224-198          | YYYY-MM-DD HH:MM:SS,### | com.ca.im.portal.common.web.util.GlobalAdminAuthInterceptor      
      | SsoToken is expired or could not be decrypted

In both the (default paths) /opt/CA/PerformanceCenter/PC/logs/PCService.log and the /opt/CA/PerformanceCenter/DM/logs/DMService.log we see this error:

ERROR | qtp341763619-246 | 2021-11-03 13:16:14,316 | com.ca.im.portal.api.services.datasource.DataSourcePoll
| Received WebServiceException from version check for data source Event Manager@<EM_Host>. CAUSE=org.apache.cxf.transport.http.HTTPException: HTTP response '401: Unauthorized' when communicating with http://<EM_Host>:8281/EventManager/DataSourceWS.asmx. MESSAGE=Could not send Message.. Returning DS_COMM_FAILURE result.
ERROR | qtp341763619-246 | 2021-11-03 13:16:14,316 | com.ca.im.portal.api.services.datasource.DataSourcePoll
| javax.xml.ws.WebServiceException: Could not send Message.

Column K is missing or set to 0 post upgrade

Environment

All supported DX NetOps Performance Management releases

Cause

The upgrade fails to copy the SsoEncryptionDecryptionKey value to the Event Manager MySql em database.

Resolution

NOTE: When prompted, you will need the Netops Portal MySQL password to enter into the MySQL CLI.

Gather and compare the SsoEncryptionDecryptionKey value between the netqosportal and em databases.
1. Connect to the MySql DB using (default path) this command. Enter the password when prompted.
  
  /opt/CA/MySql/bin/mysql -uroot -p
2. Run the following to show the value for the netqosportal DB:
  
  a) If you are on 23.3.3 or earlier run
  
  select PropValue,Priority from netqosportal.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
  
  b) If you are on 23.3.4 or later run
  
  select PropValue,Priority,K from netqosportal.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
3. Run the following to show the value for the em DB:
  
  a) If you are on 23.3.3 or earlier run:
  
  select PropValue,Priority from em.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
  
  b) If you are on 23.3.4 or later run:?
  
  select PropValue,Priority,K from em.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
The values at the highest Priority value shown should match.
1. If the values do not match for the highest Priority value shown, continue with the remaining steps below to resolve the problem.
2. If you are on 23.3.4 and later and the K values do not match continue with the remaining steps below to resolve the problem.
3. IF the values do match for the highest Priority value shown, contact support for additional assistance.
Stop the Event Manager service using the command:

systemctl stop caperfcenter_eventmanager
In the MySql prompt run the following to update the em database SsoEncryptionDecryptionKey value to match the netqosportal database value.
1. If you are on DX Netsop 23.3.3 or earlier run:
  
  replace into em.performance_center_properties values ('SsoEncryptionDecryptionKey',1,'<keyFrom_netqosportal_DB>','N',UNIX_TIMESTAMP());
2. If you are on 23.3.4 or later run:
  
  replace into em.performance_center_properties values ('SsoEncryptionDecryptionKey',1,'<keyFrom_netqosportal_DB>','N',UNIX_TIMESTAMP(),1);
3. If you are on 23.3.4 or later and the keys match but the K value is 0 in the em database query run:
  
  update em.performance_center_properties set k=1 where propname='SsoEncryptionDecryptionKey' and priority=1;
Run the following in the MySql prompt to verify the value is updated and matches in both databases.
1. Run the following to show the value for the netqosportal DB:
  
  If on 23.3.3 or earlier run:
  
  select PropValue,Priority from netqosportal.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
  
  If on 23.3.4 or later run
  
  select PropValue,Priority,K from netqosportal.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
2. Run the following to show the value for the em DB:
  
  If on 23.3.3 or earlier run:
  
  select PropValue,Priority from em.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
  
  If on 23.3.4 or later run:
  
  select PropValue,Priority,K from em.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
3. Confirm the highest Priority value from both has a matching value.
Start the Event Manager service using the command:

systemctl start caperfcenter_eventmanager

Confirm the errors are gone from the logs and the Event Manager Data Source is again successfully syncing in the Portal web UI.

This is a sample of a working system. Note the highest Priority value for the netqosportal DB (0) has a value that matches the highest Priority value (1) from the em DB. If these did not match, we'd update the em DB
Priority value 1 to match the netqosportal DB Priority value 0 value.

23.3.3 and earlier

23.3.4 and later

Additional Information

This can also happen if a local override is set for "SsoEncryptionDecryptionKey". In that scenario you will see 2 values for each query.

In that scenario where normally netqosportal only has a value set for Priority 0 we'd see it set with a Priority 1 value as well.

If that is found use the (default path) /opt/CA/PerformanceCenter/SsoConfig tool to reset the value for the SingleSign-On SsoEncryptionDecriptionKey value.