Portal Alarm Console Dashboard fails to show Spectrum Alarms

Products

DX NetOps CA Performance Management - Usage and Administration

Issue/Introduction

After a recent upgrade of the DX NetOps Portal the Alarm Console Dashboard no longer shows Spectrum Alarms.

In the (default path shown) /opt/CA/PerformanceCenter/EM/logs/EMService.log we see these errors.

ERROR | qtp513284343-3753 | 2024-08-26 13:39:22,623 | com.ca.im.portal.api.common.services.impl.database.DBPrioritizedProperties
| Error refreshing properties from database
org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [SELECT PropName, PropValue, Priority, Deleted, K FROM performance_center_properties ORDER BY PropName, Priority DESC]; nested exception is java.sql.SQLSyntaxErrorException: Unknown column 'K' in 'field list'
...
Caused by: java.sql.SQLSyntaxErrorException: Unknown column 'K' in 'field list'

Reviewing the (default path shown) /opt/CA/PerformanceCenter/InstallLogs/emmigrate.log.<DATE> log file from the latest upgrade we see these errors.

SEVERE: com.ca.im.installanywhere.util.DbMigrateException: com.ca.im.installanywhere.util.DbMigrateException: An exception occurred on step 0 (upgrade/21.2.2.0/alter.21.2.2.0.sql) of loading /apps/CA/PerformanceCenter/Tools/bin/../../EM/SQL/upgrade/21.2.2.0/_upgrade.21.2.2.0.xml
java.sql.SQLException: Error executing command: CALL copy_sso_encryption_key() on line 45 in script /apps/CA/PerformanceCenter/Tools/bin/../../EM/SQL/upgrade/21.2.2.0/alter.21.2.2.0.sql Data truncation: Data too long for column 'l_propValue' at row 2

Checking for the K column in a MySql prompt shows it is missing.

mysql> select K from em.performance_center_properties;
ERROR 1054 (42S22): Unknown column 'K' in 'field list'

Checking the Portal MySql Event Manager EM DB revision information in a MySql prompt it has not updated in many releases.

mysql> select * from em.revision_info;
+---------+---------------------+-----------------+----+
| Version | InstallDate | DbSchemaVersion | ID |
+---------+---------------------+-----------------+----+
| 2.0.0.2 | 2021-01-05 08:28:04 | 20.2.6.0 | 1 |
+---------+---------------------+-----------------+----+
1 row in set (0.01 sec)

The systems (default path shown) /opt/CA/PerformanceCenter/InstallLogs/.history file shows many other releases installed since 20.2.6 referenced by the EM DB.

Performance Center 20.2.6.513 installed at 01/05/2021 14:28:48
Performance Center 21.2.4.277 installed at 10/21/2021 08:57:48
Performance Center 21.2.8.439 installed at 03/08/2022 14:22:34
Performance Center 21.2.12.629 installed at 11/02/2022 18:25:22
Performance Center 22.2.5.1037 installed at 04/04/2023 17:44:17
DX NetOps Portal 23.3.12.1 installed at 08/26/2024 03:45:59

Environment

All supported DX NetOps Performance Management releases

Cause

Code defect results in Portal MySql Event Manager EM DB upgrade failures.

Resolution

Complete the following steps to resolve this.

Open a MySql CLI connection and run the following insert command.
- Replace <DATE> with todays date.
- The release should match the one from the emmigration log file error. Modify it as needed.
- Example for a run on August 28th, 2024:
  - insert into em.revision_info(Version, InstallDate, DbSchemaVersion) values ('2.0.0.2', '2024-08-28 00:00:00', '21.2.2.0');
Run the Portal upgrade again by running the CAPerfCenterInstall.bin command
After the install completes stop the four Portal services leaving MySql up.
- Can use this to stop the four services:
  - systemctl stop caperfcenter_*
- Does the em DB alarms table have the DescriminitorValues column present?
  - Enter a MySql CLI and run the following command.
    - desc em.alarms;
  - If we see a DiscriminatorValues column em DB upgrades up to the 22.2.6 release, the one that introduced that new column, are successful. We should see a related 22.2.6 or newer entry in the em.revision_info table.
- Does the em DB performance_center_properties table have the K column present?
  - Enter a MySql CLI and run the following command.
    - select K from em.performance_center_properties;
  - If we see a K column em DB upgrades up to the 23.3.4 release, the one that introduced that new K column, were successful. We should see a related 23.3.4 or newer entry in the em.revision_info table.
- Check release entries for em and netqosportal DBs.
  - Enter a MySql CLI and run the following commands.
    - select * from em.revision_info;
    - select * from netqosportal.revision_info;
  - Do the releases match between tables? If yes, the system is clean. No further steps are necessary.
  - If there are not matches, if the em.revision_info table is behind the netqosportal version:
    - Check the newest emmigration.log.<DATE> file in (default path) /opt/CA/PerformanceCenter/InstallLogs. Does it have errors?
    - If there are no errors, will need to insert the newest release from netqosportal.revision_info into em.revision_info, and re-run the upgrade.

The process is complete, the em DB is properly upgraded, when netqosportal.revision_info and em.revision_info DB table values match. Both should show the same release.