Email alerts and SNMP traps are not sent for repeat trigger conditions

Products

• VMware vCenter Server

Issue/Introduction

To provide a way for users to reduce the throttling of alarms by reducing the alarms reporting frequency.

Symptoms:

• Alarms are configured for email alerts or SNMP traps with state change triggers.
• Initial email alerts and SNMP traps are sent.
• Alarms are not triggered multiple times within a short time period.

Environment

VMware vCenter Server 8.0.x
VMware vCenter Server 7.0.x
VMware vCenter Server 6.7.x
VMware vCenter Server 6.5.x

Cause

The default time period for an alarm is 5 minutes and cannot be changed via the Web Client.

This means, if an alert is triggered multiple times within a 5 minute window, it will only carry out its actions once. This also results in reset rule actions being ignored if the alarm is reset within 5 minutes from when it was first triggered.

Resolution

This 5 minute frequency is expected behavior and serves as a way to prevent spamming of alarm events which could lead to an increased database size. This can result in other complications.

Workaround:
Adjust the frequency of the alarms to 0. Setting the frequency to 0 effectively disables any throttling on alarms and their actions.  

Note: Editing and saving the alarm after applying either of the workarounds below will revert the reporting frequency to its default of 5 minutes again.

To work around this issue, follow one of the options below:

Using PowerCLI

1. Download the zipped file reset-alarm-reporting-frequency.ps1.gz, that is attached to this KB. This has been provided by VMware Engineering. 
2. Open the script in a text editor of your choice and replace the text SPECIFY with the relevant info and save it.

For example:
$vcIP = 'VCENTER_IP'
$user = 'VCENTER_SSO_ADMIN_USERNAME'
$pass = 'VCENTER_SSO_ADMIN_PASSWORD'
$alarmName = "ALARM_NAME"
 

3. Run the PowerCLI script.

 

Using the Database

1. Connect to the vCenter Server database.

For more information, see Connecting to the embedded vPostgres Database in a Windows installed vCenter Server 6.x or Interacting with the vCenter Server Appliance 6.5/6.7/7.0/8.0 embedded vPostgres Database.

2. Run the script below to get the XML required to update the alarm:

SELECT setting_data FROM vpx_alarm LIMIT 1;

You will see output similar to:

<obj xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:vim25" versionId="7.0.0.0" xsi:type="AlarmSetting"><toleranceRange>0</toleranceRange><rep
ortingFrequency>300</reportingFrequency></obj>

Note: The XML contains unique versionId values for each version of vCenter Server.

 

4. Copy the XML content into the command below, change the reportingFrequency value to 0 and run the command:

UPDATE vpx_alarm SET setting_data='XML_FROM_STEP_3' WHERE name='ALARM_NAME';

For example, for vCenter Server 7.0 GA, this command would update the reportingFrequency of My Alarm to 0:

UPDATE vpx_alarm SET setting_data='<obj xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:
vim25" versionId="7.0.0.0" xsi:type="AlarmSetting"><toleranceRange>0</toleranceRange><reportingFrequency>0</reportingFrequency></obj>' WHERE name='My Alarm';
 

5. Restart the vCenter Server service using the command:

service-control --stop vmware-vpxd;service-control --start vmware-vpxd;