Performance Management Portal does not connect to DA after being upgraded from release 3.6.x/3.7.x to 21.2.1 or later

Products

DX NetOps CA Performance Management - Usage and Administration

Issue/Introduction

The Performance Management (Portal) is unable to contact the DA (Data Aggregator) after upgrading from 3.6.x or 3.7.x to 21.2.1 or a later release.

We are following the steps outlined in the following KB article:

Data Aggregator will not sync after Netops Portal upgrade to version 21.2.2 or later
https://knowledge.broadcom.com/external/article?articleId=223381

But unable the get the RESTful output via curl on the DA machine (you can replace the admin account with another one with the Administrator role).

HTTP/1.1 403 Forbidden

$ curl -u admin -v -s -k -X GET http://localhost:8581/rest/dataaggregator

$ curl -u admin -v -s -k -X GET http://localhost:8581/rest/

Environment

DX NetOps Performance Management Release : 21.2

Component : IM Data Aggregator

Cause

There were two problems:

SsoEncryptionDecryptionsKey mismatch between Portal (MySQL) and DR (Vertica)
NpcHostName has an invalid hostname in DR (Vertica)

If the NpcHostName in DA for 3.6 or 3.7 is not resolvable or reachable, we can't validate the REST credentials and allow the user to update via REST.

We have no way to update DA except via REST. But that assumes DA can reach Portal correctly.

On the DR - Vertica machine:

Gather the SsoEncryptionDecryptionsKey and NpcHostName from the Vertica database to cross-check with the Portal.

$ cd /opt/vertica/bin

Login as dauser NOT dradmin

$ ./vsql -U dauser -W (prompt for password)

 select item_id,attr_qname,string_value from dauser.v_attribute_instance where attr_qname like '%SsoEncryptionDecryptionKey';

item_id | attr_qname | string_value
---------+--------------------------------------------------------------------+--------------
8 | {http://im.ca.com/da}DataAggregatorInfo.SsoEncryptionDecryptionKey | #$utP9%z
(1 row)

 select item_id,attr_qname,string_value from dauser.v_attribute_instance where attr_qname like '%NpcHostName';

item_id | attr_qname | string_value
---------+-----------------------------------------------------+---------------
8 | {http://im.ca.com/da}DataAggregatorInfo.NpcHostName | https
(1 row)

Note down the item_id, in this case, 8. It will be used later to identify the specific DA linked attribute to be updated.

On the Portal machine:

Gather the SsoEncryptionDecryptionsKey from the MySQL database.

$ mysql -uroot -p (prompt for password)

You may have different errors in the log files:

========================================================

In the /opt/IMDataAggregator/apache-karaf-4.2.6/data/log/karaf.log file (included in the DA remote engineer output)

In the /opt/IMDataAggregator/apache-karaf-4.2.6/data/log/AuthenticationLog.log file (included in the DA remote engineer output)

The Web Service Host value is https which is not a valid hostname for the Portal machine.

Resolution

We will update the Vertica database with the SsoEncryptionDecryptionsKey value from the MySQL database and the Portal hostname (manually).

1. First, stop the DA service on the DA machine.

$ service dadaemon stop

2. Go to the DR machine and update the following attributes in Vertica. Note, use the item_id as found above in the previous select statement on the DR, in this example, 8.

$ cd /opt/vertica/bin

$ ./vsql -U dauser -W (prompt for password)

This query will update the Portal hostname and the SsoEncryptionDecryptionKey at the same time:

Please replace ‘correctPortalhostname.whatever.org’, ‘a1P4UMWW’, and 8 with the correct values for your environment.

update attribute_instance set string_value='correctPortalhostname.whatever.org' where item_id=8 and attribute_id=(select attribute_id from v_attribute where attr_qname like '%}DataAggregatorInfo.NpcHostName' limit 1);update attribute_instance set string_value='a1P4UMWW' where item_id=8 and attribute_id=(select attribute_id from v_attribute where attr_qname like '%}DataAggregatorInfo.SsoEncryptionDecryptionKey' limit 1);commit;

If only one of these values needs to be changed:

This query only updates the Portal hostname:

dauser=> update attribute_instance set string_value='Portal_hostname' where item_id=8 and attribute_id=(select attribute_id from v_attribute where attr_qname like '%}DataAggregatorInfo.NpcHostName' limit 1);

dauser=> commit;

This query only updates the SsoEncryptionDecryptionKey:

dauser=> update attribute_instance set string_value='a1P4UMWW' where item_id=8 and attribute_id=(select attribute_id from v_attribute where attr_qname like '%}DataAggregatorInfo.SsoEncryptionDecryptionKey' limit 1);

dauser=> commit;

3. Start the DA service.

$ service dadaemon start

4. The Portal successfully contacted the DA.

Additional Information

Update the SSO Encryption/Decryption Key on the Data Aggregator (21.2.6)
https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/dx-netops/21-2/Performance-Monitoring-with-DX-Performance-Management/upgrading/complete-the-upgrade.html