Threshold Events stopped appearing in DX NetOps Performance Management

Products

• CA Infrastructure Management
• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

After setting a number of Thresholds in the Monitoring Profiles (MP) Event Rules, no Threshold Events are being triggered despite many devices in the associated Groups associated to the MP attached.

The Data Aggregator shows a yellow Degraded state on the System Status page.

The Data Aggregator System Status page entry shows Threshold Monitoring is Disabled.

Environment

All supported DX NetOps Performance Management releases

Cause

When any Threshold Evaluations against a Metric Family take over 80% of a poll cycle to complete, the Threshold Monitoring engine transitions to a Degraded state.

When this takes place, a system Event is generated against the Data Aggregator noting that if Threshold Evaluations continue to take over 80% of a poll cycle, the Threshold Evaluation engine will be disabled if it persists for 15 minutes.

Resolution

Threshold evaluations are not resumed automatically after they are suspended even if the product is restarted.

Follow these steps to check and see if threshold events were shut off.

1. Enter the following information into a REST client (or WEB browser): http://<DA_host>:port/rest/thresholdmonitoring/config
   
   
   If ThresholdMonitoringLimiterEnabled is false as in the example below, then threshold monitoring was automatically shut off and you will need to reset it to resume processing threshold events
   
   <ThresholdMonitoringLimiterEnabled>false</ThresholdMonitoringLimiterEnabled>
   
   
2. To reset threshold monitoring perform the following steps. Take note of the ID value of the ThresholdMonitoringConfiguration item from the output above. For example:
   
   <ThresholdMonitoringConfigurationList>
       <ThresholdMonitoringConfiguration version="1.0.0">
          <ID>16</ID>
          <ThresholdMonitoringEnabled>false</ThresholdMonitoringEnabled>
          <PercentOfPollCycleThreshold>50</PercentOfPollCycleThreshold>
          <ThresholdMonitoringLimiterEnabled>False</ThresholdMonitoringLimiterEnabled>
          <RecoveryIntervalInMinutes>15</RecoveryIntervalInMinutes>
       </ThresholdMonitoringConfiguration>
   </ThresholdMonitoringConfigurationList>
   
   
3. Open a REST client editor or HTTP tool that sends requests and gets responses.
   
   
4. Set the Content-type to application/xml. Enter the following filter criteria: URL: http://<DA_host>:port/rest/thresholdmonitoring/config/ID
   
   where ID is the identification number that is assigned to the ThresholdMonitoringConfiguration item mentioned above, which in this example is 16.
   
   HTTP method = PUT
   
   To Resume threshold evaluations, place the following in the Body tab of the HTTP Request pane:
   
       <ThresholdMonitoringConfiguration version="1.0.0">
            <ThresholdMonitoringEnabled>true</ThresholdMonitoringEnabled>
       </ThresholdMonitoringConfiguration>
   
   Then execute the put and events should resume. The following is an example of enabling monitoring using rest from the Data Aggregator command-line using the same id as noted in the examples above:
   
   curl -v -X PUT -H "Content-Type: application/xml" -d '<ThresholdMonitoringConfiguration version="1.0.0"><ThresholdMonitoringEnabled>true</ThresholdMonitoringEnabled></ThresholdMonitoringConfiguration>' http://x.x.x.x:8581/rest/thresholdmonitoring/config/16
   
   For 21.2.x releases that require DA Authentication for REST Web services use this curl statement. Replace <UserName> and <Password> with an administrative user name and it's password.
   
   curl -v -u <UserName>:<Password> -X PUT -H "Content-Type: application/xml" -d '<ThresholdMonitoringConfiguration version="1.0.0"><ThresholdMonitoringEnabled>true</ThresholdMonitoringEnabled></ThresholdMonitoringConfiguration>' http://x.x.x.x:8581/rest/thresholdmonitoring/config/16

Additional Information

For further information on Threshold Events and Threshold Event processing see the Threshold Monitoring and Threshold Limiter Behavior documentation.