Email alerts and SNMP traps are not sent for repeat trigger conditions

Products

• VMware vCenter Server

Issue/Introduction

To provide a way for users to reduce the throttling of alarms by reducing the alarms reporting frequency.

Symptoms:

• Alarms are configured for email alerts or SNMP traps with state change triggers.
• Initial email alerts and SNMP traps are sent.
• Alarms are not triggered multiple times within a short time period.

Environment

VMware vCenter Server 6.7.x
VMware vCenter Server 7.0.x
VMware vCenter Server Appliance 6.7.x
VMware vCenter Server 6.5.x
VMware vCenter Server Appliance 6.5.x

Cause

The default time period for an alarm is 5 minutes and cannot be changed via the Web Client.

This means, if an alert is triggered multiple times within a 5 minute window, it will only carry out its actions once. This also results in reset rule actions being ignored if the alarm is reset within 5 minutes from when it was first triggered.

Resolution

This 5 minute frequency is expected behavior and serves as a way to prevent spamming of alarm events which could lead to an increased database size. This can result in other complications.

Workaround:
Adjust the frequency of the alarms to 0. Setting the frequency to 0 effectively disables any throttling on alarms and their actions.  

Note: Editing and saving the alarm after applying either of the workarounds below will revert the reporting frequency to its default of 5 minutes again.

To work around this issue, follow one of the options below:

Using PowerCLI

1. Download the zipped file reset-alarm-reporting-frequency.ps1.gz, that is attached to this KB. This has been provided by VMware Engineering. 
2. Open the script in a text editor of your choice and replace the text SPECIFY with the relevant info and save it.

For example:
$vcIP = 'VCENTER_IP'
$user = 'VCENTER_SSO_ADMIN_USERNAME'
$pass = 'VCENTER_SSO_ADMIN_PASSWORD'
$alarmName = "ALARM_NAME"
 

3. Run the PowerCLI script.

 

Using the Database

1. Connect to the vCenter Server database.

For more information, see Connecting to the embedded vPostgres Database in a Windows installed vCenter Server 6.x or Interacting with the vCenter Server Appliance 6.5/6.7/7.0 embedded vPostgres Database.

2. Run the script below to get the XML required to update the alarm:

SELECT setting_data FROM vpx_alarm LIMIT 1;

You will see output similar to:

<obj xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:vim25" versionId="7.0.0.0" xsi:type="AlarmSetting"><toleranceRange>0</toleranceRange><rep
ortingFrequency>300</reportingFrequency></obj>

Note: The XML contains unique versionId values for each version of vCenter Server.

 

4. Copy the XML content into the command below, change the reportingFrequency value to 0 and run the command:

UPDATE vpx_alarm SET setting_data='XML_FROM_STEP_3' WHERE name='ALARM_NAME';

For example, for vCenter Server 7.0 GA, this command would update the reportingFrequency of My Alarm to 0:

UPDATE vpx_alarm SET setting_data='<obj xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:
vim25" versionId="7.0.0.0" xsi:type="AlarmSetting"><toleranceRange>0</toleranceRange><reportingFrequency>0</reportingFrequency></obj>' WHERE name='My Alarm';
 

5. Restart the vCenter Server service using the command:

service-control --stop vmware-vpxd;service-control --start vmware-vpxd;