Couldn't complete encryption/decryption error in DA karaf.log

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

Unable to start the Recovery Data Aggregator after successful upgrade 21.2.2 to 21.2.4 

Recovery DA karaf.log file shows:

"WARN  | qtp1529960894-94 | 2021-11-03T12:41:55,613 | Encryption | m.portal.api.security.Encryption  215 | portal-api.common.util |       | Couldn't complete encryption/decryption due to: Error finalising cipher data: pad block corrupted"

Have tried to cycle Data Aggregator dadaemon/activemq - no good, failed to start.

The dadaemon services are running and the systemctl command shows it is Active.

This is a Disaster Recovery environment where all databases and configuration information are replicated from Primary to Disaster Recovery on a daily basis.

The SsoEncryptionDecryptionKey for all systems MUST match or these errors are seen. When checking the values they match on all Primary systems and the Disaster Recover DA. But the Disaster Recovery Portal server has different keys.

The keys are checked as follows:

1. Data Aggregator options
   1. If REST services are accessible and working go to <DA_HOST>:8581/rest/dataaggregator and note the SsoEncryptionDecryptionKey value.
   2. If REST services are not accessible or not functional open a VSql prompt on the Data Repository database and run the following query. Note the SsoEncryptionDecryptionKey  value.
      • select * from dauser.v_attribute_instance where item_id=(select item_id from dauser.v_item_facet where facet_qname like '%DataAggregatorInfo' limit 1) order by attr_qname;
2. Portal MySql DB:
   1. Connect to the MySql DB using (default path) and enter the password when prompted.
      • /opt/CA/MySql/bin/mysql -uroot -p
   2. Run the following query against the netqosportal DB:
      • select PropValue, Priority, Deleted from netqosportal.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
   3. Run the following query against the em DB:'
      • select PropValue, Priority, Deleted from em.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';

Environment

All supported DX NetOps Performance Management releases

Cause

A custom script in use is causing the problem. The script backups up the MySql DBs from Primary Portal, copies them to the Disaster Recovery Portal server and loads them.

The script was using the old insecure MySql user passwords where the current MySql DBs use new complex passwords.

Resolution

Update the custom script to use the correct new MySql passwords.