Threshold profiles don't seem to be working.

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

It doesn't look like the threshold profiles are working since we migrated to a new install of Performance Management. Threshold Monitoring is enabled per KB article: https://knowledge.broadcom.com/external/article/37241/threshold-events-stopped-appearing-in-dx.html

<?xml version="1.0"?>
<ThresholdMonitoringConfigurationList>
<ThresholdMonitoringConfiguration version="1.0.0">
<ID>3505</ID>
<ThresholdMonitoringEnabled>true</ThresholdMonitoringEnabled>
<RecoveryIntervalInMinutes>15</RecoveryIntervalInMinutes>
<ThresholdMonitoringLimiterEnabled>true</ThresholdMonitoringLimiterEnabled>
<PercentOfPollCycleThreshold>80</PercentOfPollCycleThreshold>
</ThresholdMonitoringConfiguration>
</ThresholdMonitoringConfigurationList>

Environment

All supported DX NetOps Performance Management releases

Cause

EM database is corrupt, and unable to be repaired via mysqlcheck

Resolution

Use the following steps to drop/delete the EM data base and create a new one.

NOTES:

• Default install home paths are shown for all paths referenced.
• Instructions for starting and stopping the Performance Center and MySql services are found in the Knowledge Base article:
  • How to properly restart the DX NetOps Performance Management environment

1. Remove the Event Manager (EM) Data Source (DS)
   1. Go to Administration->Data Sources->Data Sources
   2. Select the Event Manager Data Source
   3. Delete the Event Manager Data Source
2. Stop the EM service.
3. Drop the EM database.
   1. Go to /opt/CA/MySql/bin
   2. Enter the MySql prompt using: ./mysql -uroot -p
      • Enter the password when prompted
   3. Run this command to drop the EM DB: drop database em;
   4. Note: If you are unable to drop the DB due to further errors take these steps:
      1. Stop the MySql service.
      2. Stop the four Performance Center processes
      3. Delete the /opt/CA/MySql/data/em folder
      4. Start the MySql service.
4. Run the following npcshell.sh command.
   • Replace <MySql_HostName> with the host name or IP of the MySql host.
   • Replace <MySqlPasswd> with the complex MySql password. Enclose the password in single quotes if the password contains special characters.
   • /opt/CA/PerformanceCenter/Tools/bin/npcshell.sh initem --production --dbhost <MySql_HostName> --dbpass <MySqlPasswd>
5. Start four Performance Center services
6. Add the EM DS to the Portal.
   1. Go to Administration->Data Sources->Data Sources
   2. Add the Event Manager Data Source

Review Threshold configuration and see if Events begin appearing again when Threshold Rules are violated.

Additional Information

The following errors were seen when attempting to repair and drop the em database.

mysqlcheck -u root -p --auto-repair --all-databases
ERROR | pool-5-thread-1          | 2021-04-22 09:00:51,559 | com.ca.im.portal.em.dao.mysql.query.GenericBatchInsertWithReturn
      | exception while executing batch and retrieving keys (SQL: INSERT IGNORE INTO event_identifiers(ProducerID, LocalEventID, State, OccurredOn) values (?,?,?,?))
java.sql.BatchUpdateException: Can't find file: 'event_identifiers' (errno: 2 - No such file or directory)

Repairing tables
em.event_identifiers
Error    : Can't find file: 'event_identifiers' (errno: 2 - No such file or directory)
status   : Operation failed
em.event_properties
Error    : Got error 122 from storage engine
error    : Corrupt

mysql> drop database em;
ERROR 1010 (HY000): Error dropping database (can't rmdir './em', errno: 39)