Determining why vCenter is sending duplicate emails for triggered alarm
search cancel

Determining why vCenter is sending duplicate emails for triggered alarm

book

Article ID: 407557

calendar_today

Updated On:

Products

VMware vCenter Server 8.0 VMware vCenter Server

Issue/Introduction

Symptoms

The customer claims of received duplicate email notification for the same alarm triggered in the vCenter.

The alarm can be a default alarm or a custom-created alarm in the vCenter.

This KB will help in determining and validating why vCenter is sending duplicate emails for the triggered alarm in vCenter.

Log check 

In vCenter, we need to trace the journal logs. 

Live: /usr/bin/journalctl -b -0 | less  | grep -i "alarm"

Log bundle: vc_logbundle.zip/commands/journalctl_-b--0.txt

The log traces below are for an example, in this we have a situation where six emails were sent due to six configured action items.

The actual alarm, timestamp, and other environmental variables may differ.

[Alarm Generated]

Month DD 15:00:51 vcenter_fqdn vpxd[1040]: Event [61462859] [1-1] [YYYY-MM-DDT06:00:51.939336Z] [vim.event.AlarmStatusChangedEvent] [info] [] [cluster_name] [61462859] [Alarm 'alarm_name_test' on object_name changed from Green to Red]

[Action Item]
Month DD 15:00:51 vcenter_fqdn vpxd[1040]: Event [61462860] [1-1] [YYYY-MM-DDT06:00:51.939441Z] [vim.event.AlarmActionTriggeredEvent] [info] [] [cluster_name] [61462860] [Alarm 'alarm_name_test' on object_name triggered an action]
Month DD 15:00:51 vcenter_fqdn vpxd[1040]: Event [61462861] [1-1] [YYYY-MM-DDT06:00:51.939498Z] [vim.event.AlarmActionTriggeredEvent] [info] [] [cluster_name] [61462861] [Alarm 'alarm_name_test' on object_name triggered an action]
Month DD 15:00:51 vcenter_fqdn vpxd[1040]: Event [61462862] [1-1] [YYYY-MM-DDT06:00:51.939532Z] [vim.event.AlarmActionTriggeredEvent] [info] [] [cluster_name] [61462862] [Alarm 'alarm_name_test' on object_name triggered an action]
Month DD 15:00:51 vcenter_fqdn vpxd[1040]: Event [61462863] [1-1] [YYYY-MM-DDT06:00:51.93956Z] [vim.event.AlarmActionTriggeredEvent] [info] [] [cluster_name] [61462863] [Alarm 'alarm_name_test' on object_name triggered an action]
Month DD 15:00:51 vcenter_fqdn vpxd[1040]: Event [61462864] [1-1] [YYYY-MM-DDT06:00:51.93958Z] [vim.event.AlarmActionTriggeredEvent] [info] [] [cluster_name] [61462864] [Alarm 'alarm_name_test' on object_name triggered an action]
Month DD 15:00:51 vcenter_fqdn vpxd[1040]: Event [61462865] [1-1] [YYYY-MM-DDT06:00:51.939602Z] [vim.event.AlarmActionTriggeredEvent] [info] [] [cluster_name] [61462865] [Alarm 'alarm_name_test' on object_name triggered an action]

[Emails being sent for each action]
Month DD 12:12:42 vcenter_fqdn vpxd[1040]: Event [61300802] [1-1] [YYYY-MM-DDT03:12:42.037048Z] [vim.event.AlarmEmailCompletedEvent] [info] [] [cluster_name] [61300802] [Alarm 'alarm_name_test' on object_name sent email to [email protected]]
Month DD 12:12:42 vcenter_fqdn vpxd[1040]: Event [61300803] [1-1] [YYYY-MM-DDT03:12:42.102061Z] [vim.event.AlarmEmailCompletedEvent] [info] [] [cluster_name] [61300803] [Alarm 'alarm_name_test' on object_name sent email to [email protected]]
Month DD 12:12:42 vcenter_fqdn vpxd[1040]: Event [61300805] [1-1] [YYYY-MM-DDT03:12:42.150166Z] [vim.event.AlarmEmailCompletedEvent] [info] [] [cluster_name] [61300805] [Alarm 'alarm_name_test' on object_name sent email to [email protected]]
Month DD 12:12:42 vcenter_fqdn vpxd[1040]: Event [61300804] [1-1] [YYYY-MM-DDT03:12:42.150166Z] [vim.event.AlarmEmailCompletedEvent] [info] [] [cluster_name] [61300804] [Alarm 'alarm_name_test' on object_name sent email to [email protected]]
Month DD 12:12:42 vcenter_fqdn vpxd[1040]: Event [61300806] [1-1] [YYYY-MM-DDT03:12:42.200296Z] [vim.event.AlarmEmailCompletedEvent] [info] [] [cluster_name] [61300806] [Alarm 'alarm_name_test' on object_name sent email to [email protected]]
Month DD 12:12:42 vcenter_fqdn vpxd[1040]: Event [61300807] [1-1] [YYYY-MM-DDT03:12:42.200292Z] [vim.event.AlarmEmailCompletedEvent] [info] [] [cluster_name] [61300807] [Alarm 'alarm_name_test' on object_name sent email to [email protected]]
 
UI validation
Select the affected alarm and expand the view, and you will notice additional trigger action.
 

Environment

VMware vCenter Server 7.0.x
VMware vCenter Server 8.0.x

Resolution

Validation of the triggered action from vCenter DB

Two tables help us validate the duplicate action items for the affected alarm.

  • vpx_alarm: From here, we get the alarm_id of the affected alarm we are investigating.
  • vpx_alarm_action:This table helps clarify the action items and their corresponding trigger points.

Take an SSH session to the vCenter. 

To log in to the VCDB of the VC, run this command in the SSH session: /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

Run the command below to check the alarm_id of the affected alarm

select alarm_id,name,action_frequency from vpx_alarm where name='alarm_name';

Example:

VCDB=# select alarm_id,name,action_frequency from vpx_alarm where name='VM Deployed';
 alarm_id |    name     | action_frequency
----------+-------------+------------------
      301 | VM Deployed |             7200
(1 row)

From the alarm_id Captured from the above output, we will now check the vpx_alarm_action table to understand the trigger points and the action type.

Command: select alarm_id,action_id,green2yellow_repeat_flg,red2yellow_repeat_flg,yellow2green_repeat_flg,yellow2red_repeat_flg from vpx_alarm_action where alarm_id='id';

Example 

select alarm_id,action_id,green2yellow_repeat_flg,red2yellow_repeat_flg,yellow2green_repeat_flg,yellow2red_repeat_flg from vpx_alarm_action where alarm_id='8113';

From the above output, we can confirm that the affected alarm has additional action items that were configured by an external script or any other external factor. Actions are triggered when the alarm goes from yellow to red(refer to field yellow2red).

This is the reason why we receive duplicate emails from the affected alarm.

Note: In the vCenter UI, we cannot duplicate action items; even if done, the UI will automatically delete/void the duplicate action.

vCenter MOB page validation

Once you have the alarm_id identified from the VCDB, you can validate the same action items via the vCenter MOB page. The output would match the details of the vpx_alarm_action table we investigated in the previous steps.

To do that, you can load these URLs:

    • https://<vc-ip/fqdn>/mob/?moid=alarm-id&doPath=info.action

    • https://<vc-ip/fqdn>/mob/?moid=alarm-id&doPath=info.action.action

Example:

Once the validation is done and the triggered cause is identified, please edit the external script that created these duplicate actions or reconfigure the alarm in the vCenter. This shall resolve the issue. 

Select the affected alarm, disable and then enable the alarm in the vCenter Ui. 

OR click on EDIT and remove the duplicate actions from the rule.

Powered by Wolken Software