Manually delete notification rules

Products

• VMware Aria Suite

Issue/Introduction

• When navigating to Configure > Alerts > Notifications, the page does not load, or you see an error similar to:

  Alert Definition Ids '[AlertDefinition-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx]' not found.

• New notifications cannot be created and current notifications cannot be accessed or seen.

Environment

VMware Aria Operations 8.x (formerly vRealize Operations)

Cause

This issue occurs when an existing notification rule references a deleted item such as an alert definition or an object.

Resolution

To resolve this issue, delete the offending notification rule(s) through the API or directly through the database.

• API
• Database - Aria Operations 8.10.x and higher
• Database - vRealize Operations 8.6.x and below

Impact/Risks: Modifying the database can be harmful if done incorrectly.
You MUST create a snapshot before proceeding: Snapshot Creation in VMware Aria Operations.

 

NOTE: In the text below, "offending notification rule" refers to the notification rule(s) that's causing the issue.  This is typically a rule that references an exact thing that no longer exists, such as an object or Alert Definition.  This is most commonly seen after upgrading without addressing the warnings in the Pre-Upgrade Readiness Assessment Tools.

If you do not know the ID of the offending notification rule, delete notification rules one by one and refresh the Notifications page in the UI after each deletion. Once the UI is able to load without issue you can restore from snapshot and try to more selectively delete the IDs. In some cases, all rules must be deleted.

API

1. Navigate to Home > Developer Center and click API Documentation.
2. Click the green Authorize button on the right of the page.
3. If presented with an Authorized user, click Logout
4. Enter the credentials for a user with rights to delete objects and click Authorize, then click Close.
5. Expand Notifications /api/notifications
6. Expand GET /api/notifications/rules
7. Click Try it out.
8. Click Execute.
9. In the Response body field, copy the ID value next to the "id" property for the offending notification rule(s).
   
   Example:

    "rules": [
         "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx",
         "name": "Sample notification rule name",
         "pluginId":  ...
   

   The value xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxxx is the ID of this notification rule.
   
   Tip: If you received an error when browsing to Configure > Alerts > Notifications, use the browser's search function to search the Response Body for the object ID that was presented in the error message to identify the Notification Rule(s) to delete. Scroll up from any match to find the ID of the rule that is referencing the object.
10. Expand DELETE /api/notifications/rules/{id} and click Try it out
11. Paste the ID of the offending notification rule copied in step 9, then click Execute.
12. Repeat steps 9 thru 11 if there are more offending notification rules to delete.

Database

Aria Operations 8.10.x and higher

1. Log into the Primary node as root via SSH or Console.
2. Log into the postgres repl database:

   su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -d vcopsdb -p 5433"

3. Get a list of all notification rules in the table and note the ID of the offending notification rule(s):

   select key from kv_notificationplugin where key like 'NotificationRule%'; 

4. Delete the offending notification rule:

   delete from kv_notificationplugin where key='ID';

   Note: Replace ID with the ID of the notification rule noted in step 3.
   
   Example:

   delete from kv_notificationplugin where key='NotificationRule.xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx';

   Tip: To delete multiple notification rules at once, use a comma separated list of IDs in the command:

   delete from kv_notificationplugin where key IN ('NotificationRule.ID1', 'NotificationRule.ID2', NotificationRule.ID3');

5. Repeat steps 1-4 on The Replica node if HA or CA is enabled.

 

vRealize Operations 8.6.x and below

1. Log into the Primary node as root via SSH or Console.
2. Create a list of notification rules from the notificationplugin table in Cassandra:

   $VMWARE_PYTHON_BIN $ALIVE_BASE/cassandra/apache-cassandra-*/bin/cqlsh.py --ssl --cqlshrc $ALIVE_BASE/user/conf/cassandra/cqlshrc -e 'PAGING OFF; SELECT * from globalpersistence.notificationplugin' > /tmp/notifications.txt

3. Open the /tmp/notifications.txt file using a text editor and note the ID of the offending notification rule.

   Example notification rule entry:

   notificationplugin | notificationplugin | NotificationRule.xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx | null | NotificationRule | null | null | {"NotificationRule":{"id":{"@UUID":"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","@ObjectType":"NOTIFICATION_RULE"},...

   The ID of the notification rule for this example is xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.

4. Log in to the Cassandra Database:

   $VMWARE_PYTHON_BIN $ALIVE_BASE/cassandra/apache-cassandra-*/bin/cqlsh --ssl --cqlshrc $ALIVE_BASE/user/conf/cassandra/cqlshrc

5. Delete the notification rule:

   DELETE from globalpersistence.notificationplugin where namespace='notificationplugin' and classtype='notificationplugin' and key='ID';

   Note: Replace ID with the ID of the notification rule noted in step 3.

   Example:  

   DELETE from globalpersistence.notificationplugin where namespace='notificationplugin' and classtype='notificationplugin' and key='NotificationRule.xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx';

Additional Information

Impact/Risks:
Modifying the database can be harmful if done incorrectly.
You MUST create a snapshot before proceeding: How to take a Snapshot of vRealize Operations.