Performance Management Portal does not connect to DA after being upgraded from release 3.6.x/3.7.x to 21.2.1 or later
Article ID: 231200
Updated On:
Products
Issue/Introduction
The Performance Management (Portal) is unable to contact the DA (Data Aggregator) after upgrading from 3.6.x or 3.7.x to 21.2.1 or a later release.
We are following the steps outlined in the following KB article:
Data Aggregator will not sync after Netops Portal upgrade to version 21.2.2 or later
https://knowledge.broadcom.com/external/article?articleId=223381
But unable the get the RESTful output via curl on the DA machine (you can replace the admin account with another one with the Administrator role).
HTTP/1.1 403 Forbidden
$ curl -u admin -v -s -k -X GET http://localhost:8581/rest/dataaggregator
$ curl -u admin -v -s -k -X GET http://localhost:8581/rest/
Environment
DX NetOps Performance Management Release : 21.2
Component : IM Data Aggregator
Cause
There were two problems:
- SsoEncryptionDecryptionsKey mismatch between Portal (MySQL) and DR (Vertica)
- NpcHostName has an invalid hostname in DR (Vertica)
If the NpcHostName in DA for 3.6 or 3.7 is not resolvable or reachable, we can't validate the REST credentials and allow the user to update via REST.
We have no way to update DA except via REST. But that assumes DA can reach Portal correctly.
On the DR - Vertica machine:
Gather the SsoEncryptionDecryptionsKey and NpcHostName from the Vertica database to cross-check with the Portal.
$ cd /opt/vertica/bin
Login as dauser NOT dradmin
$ ./vsql -U dauser -W (prompt for password)
select item_id,attr_qname,string_value from dauser.v_attribute_instance where attr_qname like '%SsoEncryptionDecryptionKey';
item_id | attr_qname | string_value
---------+--------------------------------------------------------------------+--------------
8 | {http://im.ca.com/da}DataAggregatorInfo.SsoEncryptionDecryptionKey | #$utP9%z
(1 row)
select item_id,attr_qname,string_value from dauser.v_attribute_instance where attr_qname like '%NpcHostName';
item_id | attr_qname | string_value
---------+-----------------------------------------------------+---------------
8 | {http://im.ca.com/da}DataAggregatorInfo.NpcHostName | https
(1 row)
Note down the item_id, in this case, 8. It will be used later to identify the specific DA linked attribute to be updated.
On the Portal machine:
Gather the SsoEncryptionDecryptionsKey from the MySQL database.
$ mysql -uroot -p (prompt for password)
mysql> select PropValue, Priority, Deleted from netqosportal.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
+-----------+----------+---------+
| PropValue | Priority | Deleted |
+-----------+----------+---------+
| a1P4UMWW | 0 | N | (current SsoEncryptionDecryptionsKey)
+-----------+----------+---------+
1 row in set (0.00 sec)
mysql> select PropValue, Priority, Deleted from em.performance_center_properties where PropName = 'SsoEncryptionDecryptionKey';
+-----------+----------+---------+
| PropValue | Priority | Deleted |
+-----------+----------+---------+
| #$utP9%z | 0 | N | (legacy SsoEncryptionDecryptionsKey)
| a1P4UMWW | 1 | N | (current SsoEncryptionDecryptionsKey)
+-----------+----------+---------+
2 rows in set (0.00 sec)
You may have different errors in the log files:
========================================================
In the /opt/IMDataAggregator/apache-karaf-4.2.6/data/log/karaf.log file (included in the DA remote engineer output)
WARN | qtp1504420869-95 | 2021-12-23T04:54:34,702 | Encryption | m.portal.api.security.Encryption 215 | portal-api.common.util | | Couldn't complete encryption/decryption due to: Error finalising cipher data: pad block corrupted
In the /opt/IMDataAggregator/apache-karaf-4.2.6/data/log/AuthenticationLog.log file (included in the DA remote engineer output)
INFO | qtp1504420869-95 | 2021-12-23T04:54:34,696 | AuthenticationLog | entication.impl.DaBaseServiceIml 94 | ager.core.authentication | | Portal hostname is https
INFO | qtp1504420869-87 | 2021-12-23T04:54:41,728 | AuthenticationLog | entication.impl.DaBaseServiceIml 94 | ager.core.authentication | | Portal hostname is https
The Web Service Host value is https which is not a valid hostname for the Portal machine.
Resolution
We will update the Vertica database with the SsoEncryptionDecryptionsKey value from the MySQL database and the Portal hostname (manually).
1. First, stop the DA service on the DA machine.
$ service dadaemon stop
2. Go to the DR machine and update the following attributes in Vertica. Note, use the item_id as found above in the previous select statement on the DR, in this example, 8.
$ cd /opt/vertica/bin
$ ./vsql -U dauser -W (prompt for password)
This query will update the Portal hostname and the SsoEncryptionDecryptionKey at the same time:
Please replace ‘correctPortalhostname.whatever.org’, ‘a1P4UMWW’, and 8 with the correct values for your environment.
update attribute_instance set string_value='correctPortalhostname.whatever.org' where item_id=8 and attribute_id=(select attribute_id from v_attribute where attr_qname like '%}DataAggregatorInfo.NpcHostName' limit 1);update attribute_instance set string_value='a1P4UMWW' where item_id=8 and attribute_id=(select attribute_id from v_attribute where attr_qname like '%}DataAggregatorInfo.SsoEncryptionDecryptionKey' limit 1);commit;
If only one of these values needs to be changed:
This query only updates the Portal hostname:
dauser=> update attribute_instance set string_value='Portal_hostname' where item_id=8 and attribute_id=(select attribute_id from v_attribute where attr_qname like '%}DataAggregatorInfo.NpcHostName' limit 1);
dauser=> commit;
This query only updates the SsoEncryptionDecryptionKey:
dauser=> update attribute_instance set string_value='a1P4UMWW' where item_id=8 and attribute_id=(select attribute_id from v_attribute where attr_qname like '%}DataAggregatorInfo.SsoEncryptionDecryptionKey' limit 1);
dauser=> commit;
3. Start the DA service.
$ service dadaemon start
4. The Portal successfully contacted the DA.
Additional Information
Feedback
