Event Manager shows synchronization failure in DX NetOps

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

After upgrading lab I am getting the Event Manager is failing to sync. Upgraded to 21.2.6.

When we review the MySql database process list we see a very long running DELETE command for the em database used by Event Manager.

It's seen running the command (default path shown) as shown. Enter your password when prompted.

./mysql -uroot -p -A -e "show full processlist\G;"

A sample entry might have the following details. We see Query ID 487 has been running for 247306 seconds per the Time value. We see it's a query against the em database that Event Manager uses and we see it's a DELETE query.

Id: 487
User: netqos
Host: localhost:42654
db: em
Command: Query
Time: 247306
State: Sending data
Info: DELETE alarms, alarm_item_properties, alarm_item_gc_memberships, alarm_filter_attribute_values FROM alarms INNER JOIN alarm_item_properties ON alarms.SourceID = alarm_item_properties.SourceID LEFT JOIN alarm_item_gc_memberships ON alarms.SourceID = alarm_item_gc_memberships.SourceID LEFT JOIN alarm_filter_attribute_values ON alarms.SourceID = alarm_filter_attribute_values.SourceID WHERE alarms.SourceID = 4

Environment

Reported against DX NetOps Performance Management release r21.2.5. May impact all available releases.

Cause

This is due to a known defect that causes Event Manager to believe it's Data Source was deleted. As a result it's trying to delete alarms to complete the removal. The query fails and hangs because the Data Source was not actually deleted.

Resolution

This is being addressed via defect DE526333. This defect has been resolved in the 21.2.8 NetOps release. Upgrade to that release or newer to obtain a permanent resolution.

Until able to upgrade the issue is resolved by killing the hung query using it's ID. We'd take the following steps using the sample above.

1. Open a terminal on the PC Server.
2. Run the following to determine and validate the correct query ID to kill. Enter the password when prompted.
   1. ./mysql -uroot -p -A -e "show full processlist\G;"
3. Run the following to kill the problem query. Replace <QueryID> with the query ID for the problem DELETE query.
   1. ./mysql -uroot -p -A -e "kill <QueryID>;"
   2. Using the above sample with Query ID 487 the command would be:
      1. ./mysql -uroot -p -A -e "kill 487;"