Smarts SAM: Escalation policy will not complete when notified; TERMINATED ESCALATION is sent. Do all source domains need to be mentioned in an escalation policy?

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:


Escalation policy triggers a "TERMINATED ESCALATION" once a unknown source is seen with a matching notification. 

 

In this example, the escalation policy was configured to trigger only off one domain 5112- AMPM01 but we see two notifications come through 4 minutes apart. The first one from the domain named 51223-AMPM01, and the second one from the domain named ESM.  Because the second domain is NOT mentioned in the policy, the code prevents the escalation policy from completing. 

Here's the escalation policy, as seen in the DMAC. Note there is only one source domain in the policy- 5112- AMPM01.



A:  Here are the two notifications we see coming from with the Source attribute from two different domains. ( The domains are in bold font ) 

• 1442384664 September 16, 2015 1:24:24 AM CDT NOTIFICATION-FileSystem_PerformanceHostResources_I-FileSystemPerformanceHostResources-FS-KZDL1020/11_5112_HighUtilization FileSystem_Performance_HostResources I-FileSystem_Performance_HostResources-FS-KZDL1020/11_5112 HighUtilization 0 DXA NOTIFY Server:5112- AMPM01
• 1442384937 September 16, 2015 1:28:57 AM CDT NOTIFICATION-FileSystem_PerformanceHostResources_I-FileSystemPerformanceHostResources-FS-KZDL1020/11_5112_HighUtilization FileSystem_Performance_HostResources I-FileSystem_Performance_HostResources-FS-KZDL1020/11_5112 HighUtilization 6 DXA SOURCE-NOTIFY Server: ESM
• 1442384937 September 16, 2015 1:28:57 AM CDT NOTIFICATION-FileSystem_PerformanceHostResources_I-FileSystemPerformance_HostResources-FS-

B: While the escalation timer is triggered on the first notification above, it is subsequently cancelled on the second notification from the ESM domain -ESM ( The termination signal is in bold below) 

•  1442384937 September 16, 2015 1:28:57 AM CDT NOTIFICATION-FileSystem_PerformanceHostResources_I-FileSystemPerformanceHostResources-FS-KZDL1020/11_5112_HighUtilization FileSystem_Performance_HostResources I-FileSystem_Performance_HostResources-FS-KZDL1020/11_5112 HighUtilization 7 SYSTEM ESCALATION TERMINATED ESCALATION: Unmatched GeneralPerformance/HoldDownTimer
•  
•  

Environment

VMware Smart Assurance - SMARTS

Cause

This software is working as designed. Here is the reason why:

1:  In ICS_AutoActionPolicy::evaluateChange() of ICS_AutoAction_Impl.c, when notification change in the notification is noted (such as it now appears to be coming from a different source ), we re-evaulate  the policy with this new notification and realize it no longer matches  the policy ( e.g., the escalation path policy).  
2: Because it no longer matches the criteria the prescheduled escalation auto-action task item queued for this path is then  be removed from the scheduler, which is what we observed here.

 

Resolution

There are two solutions to this issue:

Option 1: Any source domain that you will see the notification from needs to be mentioned in the escalation policy.

In the screen shot below, you'll see the highlighted Source domain for "5112-AMPM01"

Add a similar Source entry for each of your domains.



Option 2: Do not mention any domains in the escalation policy. This will allow the escalation policy to match a notification from any domain.
In other words, if you have a Source Domain entry as in the above example- remove it, so there is no longer a Source Domain mentioned in the policy.