Duplicate PM Threshold Violation based Alarms in Spectrum

Products

• CA Performance Management - Usage and Administration
• DX NetOps

Issue/Introduction

Seeing duplicate Alarms in Spectrum from DX NetOps Performance Management (PM) Threshold Violation Events.

The environment is configured to always only send PM Events as Traps to Spectrum SpectroServer(s) using a PM Notification Rule. This way they go through the Spec South Bound Gateway (SBG), getting asserted on the device model.

Without that, due to naming issues in PM related to the information the devices provide, the alarms show against generic interface names.

We have never had the Event Integration enabled in the Spectrum OneClick (OC) web server. It is confirmed as Disabled on all Spectrum OC systems in the environment.

Despite this, we're seeing duplicate Alarms from the same PM Event, one via integration sync and one via trap.

Environment

All supported DX NetOps Performance Management releases

Cause

Unable to determine why Spectrum was polling PM for Events despite the Event Integration being disabled on the OC server.

Resolution

There are two options to resolve this:

• Stop and restart the OC tomcat web server services. This should set the Disabled state for the Event Integration and halt the behavior.
• Set the Event Integration to Enabled. Wait 20 seconds. Set the Integration to Disabled. This should ensure the Disabled state is set properly for the Event Integration and halt the behavior.

Additional Information

Logs, and log messaging, from the time frame when the issue began are no longer available. Due to that we are unable to determine a specific root case.

Spectrum will ONLY will poll PM for Events when the Event Integration has been enabled. 

Nothing in the PM or Spectrum code would enable or disable the Event Integration on it's own. When Spectrum OC web server services are started, unless the Event Integration is set to Enabled, it doesn't build a thread in the system to start the Event Polling. That is only done when the integration is set to Enabled. When the integration is set to disabled, that Event Polling thread for the integration is closed.

The best educated guess we can make is that a user, or user initiated activity like a script, mistakenly set the Integration to Enabled. The mistake was realized and it was quickly set to Disabled. If the timing is right, if it was sent a Disable state change while it was still in the process of enabling itself and starting its polling thread, it might have failed to Disable it. The thread might have started AFTER the disabled state change arrived, before it was seen and ignored since no thread had actually started yet. That could leave it in the state found here, where it's seen in the UI as disabled, yet it is actively polling PM for Events.