Understanding the different EventState values of Notifications in Smarts SAM Domain.

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:

Understanding the different EventState values of Notifications in SAM.

Environment

SMARTS:10.1.x

Resolution

• When a notification is created (notified) in SAM, the direct source that notified it is marked as an ACTIVE source for that notification. 
• A source is one whose name is passed along with notify() function.  There may be multiple sources that report an event in different states (ACTIVE, SUSPENDED, INACTIVE).  The notification state becomes WAS_ACTIVE when a running ACTIVE source disconnects from SAM.

(Note that a source need not be a running InCharge domain server; rather it is the name of a source that is known to report about a given event or notification.) 

• So the trap manager ("Trap Adapter"), or syslog manager ("Syslog Adapter"), or a source that is specified in sm_ems command are direct sources of OI/SAM notifications.  In case of DXA, the source is a domain manager, which can be AM-PM, OI, or another SAM.
• It is responsibility of individual adapter or program to update the state of a notification that is previously notified by it when changes occur.  SAM treats both types of sources (a real domain or a named source) the same way:  If the last state was ACTIVE for a given source, it will be changed to WAS_ACTIVE if:
        a) SAM is restarted
        b) SAM is disconnected from the domain server. 
• In either case, all the sources for all notifications are marked as WAS_ACTIVE.  The state of a source of a previously active notification remains WAS_ACTIVE so long as the source provides an update.  The lack of an update is interpreted as CLEAR event only in the case when SAM is reconnected to the InCharge domain manager via DXA after a waiting period.
• So, if a domain remains disconnected, or a non-running source does not clear or re-notify the event, WAS_ACTIVE state is maintained forever.  If it is a domain and will no longer run, it should be removed from ics.conf.  If it is an sm_ems source, sm_ems should either clear or re-notify the notification after SAM becomes available again.  In the case of the Trap or Syslog Adapters, a clear or re-notify of the trap or syslog message will correct the state of the notification.  Otherwise, it will remain WAS_ACTIVE.  When OI has a notification as WAS_ACTIVE, SAM suspends it indicating that no updates regarding the notification are currently available (from its sources).
• The best approach to avoid these situations is to assign an expiration to Trap or Syslog Adapter sourced notifications.  Expiration should not be so short as to generate multiple notifications.  Notifications generated by sm_ems are custom and have to be updated when SAM is restarted.