Event Manager failed to upgrade
Article ID: 265673
Updated On:
Products
Issue/Introduction
Event Manager failed to upgrade
Alarm Console Dashboard shows No Data to Display post upgrade
Performance Management Alarms console no longer getting alarms from Spectrum
No Spectrum alarm in PC/Portal web page
MySql errors after upgrading from 21.2.10 or earlier releases to 21.2.11 or later releases.
Error found in the EM/logs/wrapper-<DATE>.log file:
The em DB is not updated with new release details (still in 3.7.8.0) while the netqosportal DB is (22.2.6.0).
Environment
All supported DX NetOps Performance Management releases between 21.2.11 (when the merge_old_partitions process at issue is introduced) and 23.3.9 when the issue is expected to be resolved (ETA Subject to change).
Cause
The EM MySql DB was not successfully upgraded
Resolution
-
To resolve this complete the following steps.
The steps will bring the DB up the the correct level progressively.
The following npcshell.sh commands require special steps.
- All environments and systems with MySql and Portal services on the same host:
- Update the DB_PASSWORD with your MySql DB password.
- If MySql runs on it's own host you will also need to:
- Run the npcshell commands on the Portal host.
- Edit the --dbhost value replacing localhost with the IP or name of the MySql host.
- Stop the four Portal services (caperfcenter_console, caperfcenter_devicemanager, caperfcenter_eventmanager and caperfcenter_sso). Leave MySql running. E.G. systemctl stop caperfcenter_*
- Run the following command.
- /opt/CA/PerformanceCenter/Tools/bin/npcshell.sh upgradeem --production --dbhost localhost --dbport 3306 --dbuser 'netqos' --dbpass 'DB_PASSWORD' --log /opt/CA/PerformanceCenter/InstallLogs/emmigrate.log.`date +%F_%H-%M`
- Note the single quotes around the date portion.
- It will return errors. That is expected.
- The errors will be from the run of a sql script named merge_old_event_partitions.sql.
- /opt/CA/PerformanceCenter/Tools/bin/npcshell.sh upgradeem --production --dbhost localhost --dbport 3306 --dbuser 'netqos' --dbpass 'DB_PASSWORD' --log /opt/CA/PerformanceCenter/InstallLogs/emmigrate.log.`date +%F_%H-%M`
- Run the following command, from the /opt/CA/MySql/bin folder:
- ./mysql -uroot -p em < /opt/CA/PerformanceCenter/EM/SQL/event_partitions.sql
- Open a MySql CLI connection and run the following insert command. Replace <DATE> with todays date.
- insert into em.revision_info(Version, InstallDate, DbSchemaVersion) values ('2.0.0.2', '<DATE> 00:00:00', '21.2.11.0');
- Example for a run on May 11th, 2023:
- insert into em.revision_info(Version, InstallDate, DbSchemaVersion) values ('2.0.0.2', '2023-05-11 00:00:00', '21.2.11.0');
- Run the npcshell upgradeem command again.
- /opt/CA/PerformanceCenter/Tools/bin/npcshell.sh upgradeem --production --dbhost localhost --dbport 3306 --dbuser 'netqos' --dbpass 'DB_PASSWORD' --log /opt/CA/PerformanceCenter/InstallLogs/emmigrate.log.`date +%F_%H-%M`
- It will return errors. This is expected.
- The errors will be from the run of a sql script named merge_old_event_partitions.sql.
- Depending on DB and table sizes, the add_pks_to_em_tables.sql command issued may take 10-15+ minutes to run.
- Check em.revision_info once again. If the listed revision is 23.3.x+, check the latest emmigrate.log for any additional errors. If the revision is 23.3.x+ and there are no further errors in the emmigrate.log, you are done; proceed to step 9.
- select * from em.revision_info
- Open a MySql CLI connection and run the following insert command. Replace <DATE> with todays date.
- insert into em.revision_info(Version, InstallDate, DbSchemaVersion) values ('2.0.0.2', '<DATE> 00:00:00', '22.2.2.0');
- Example for a run on May 11th, 2023:
- insert into em.revision_info(Version, InstallDate, DbSchemaVersion) values ('2.0.0.2', '2023-05-11 01:00:00', '22.2.2.0');
- Run the npcshell upgradeem command again.
- /opt/CA/PerformanceCenter/Tools/bin/npcshell.sh upgradeem --production --dbhost localhost --dbport 3306 --dbuser 'netqos' --dbpass 'DB_PASSWORD' --log /opt/CA/PerformanceCenter/InstallLogs/emmigrate.log.`date +%F_%H-%M`
- It should not return errors on this run. Errors are not expected for this run of the command.
- When that completes to validate the solution run the following. There should now be a DescriminatorValues column added to the table.
- Open a MySql CLI connection and run the following command.
- desc em.alarms;
- Open a MySql CLI connection and run the following command.
Restart the four PC web server services if the missing column is present.
E.G. systemctl start caperfcenter_sso; systemctl start caperfcenter_devicemanager; systemctl start caperfcenter_eventmanager; sleep 30; systemctl start caperfcenter_console;
The Alarms Console should now work properly.
- All environments and systems with MySql and Portal services on the same host:
Additional Information
This issue will only be seen in systems that are upgraded to 21.2.11 through 23.3The issue first begins with 21.2.11 and newer installations. This is the first release to include the merge_old_partitions MySql script that causes the issue.
Only upgrades to 22.2.6 or newer will see the missing DiscriminatorValues DB value. It was first introduced with the 22.2.6 release.
mysql> select * from em.revision_info;
+---------+---------------------+-----------------+----+
| Version | InstallDate | DbSchemaVersion | ID |
+---------+---------------------+-----------------+----+
| 2.0.0.2 | 2016-07-20 18:30:46 | 2.5.0.0 | 1 |
| 2.0.0.2 | 2017-06-20 15:04:06 | 3.1.0.0 | 2 |
| 2.0.0.2 | 2018-07-12 10:03:06 | 3.5.0.0 | 3 |
| 2.0.0.2 | 2019-08-29 15:49:29 | 3.7.3.0 | 4 |
| 2.0.0.2 | 2020-01-29 19:23:18 | 3.7.8.0 | 5 |
| 2.0.0.2 | 2023-05-09 00:00:00 | 21.2.11.0 | 6 |
| 2.0.0.2 | 2023-05-09 00:00:00 | 22.2.2.0 | 7 |
| 2.0.0.2 | 2023-05-09 18:38:39 | 22.2.6.0 | 8 |
+---------+---------------------+-----------------+----+
8 rows in set (0.00 sec)
Feedback
