WARNING messages “Event timeout has been reached.” in Event Processing Manager Log

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:

Following WARNING messages in the emc-smarts Event Processing Manager logs reporting an  Event timeout  at regular intervals.
 
WARNING -- [2016-06-05 10:38:10 GST] -- a::validate(): Event timeout has been reached. Reconnecting to source and resynchronizing events.
INFO -- [2016-06-05 10:38:10 GST] -- a::disconnect(): Disconnecting from myserver.emc.com:426:INCHARGE-SA-PRES.
INFO -- [2016-06-05 10:38:10 GST] -- a::connect(): Connecting to myserver.emc.com:426: INCHARGE-SA-PRES...
INFO -- [2016-06-05 10:38:11 GST] -- a::connect(): Connection successful!
INFO -- [2016-06-05 10:38:11 GST] -- a::resync(): Starting resynchronization for source INCHARGE-SA-PRES...
INFO -- [2016-06-05 10:38:11 GST] -- OccurrenceStoreResynchronizer::mark(): Marking source  INCHARGE-SA-PRES ' with date 1465108691034
INFO -- [2016-06-05 10:38:11 GST] -- OccurrenceStoreResynchronizer::sync(): Synching source INCHARGE-SA-PRES ' with mark 1465108691034
INFO -- [2016-06-05 10:38:11 GST] -- a::resync(): Resync operation completed on myserver.emc.com:426   INCHARGE-SA-PRES for ICS_NL-Default (50 events synchronized).
 

Environment

Watch4Net/M&R - 7.x

Cause

The Event timeout period referred to in the WARNING message is as specified in the <idle-timeout> parameter below.  By default this is timeout set to 5 minutes.


        <resync-period>12h</resync-period>
        <connection-check-period>30s</connection-check-period>
        <connection-timeout>60s</connection-timeout>
        <idle-timeout>5m</idle-timeout>
        <source id="INCHARGE-SA-PRES">
                <primary id="primary">
                        <broker-hostname>localhost</broker-hostname>
                        <broker-port>426</broker-port>
                        <domain-name>INCHARGE-SA-PRES</domain-name>
                        <username>admin</username>
                        <password>{<encrypted_password>}</password>
                        <notification-list>ICS_NL-Default</notification-list>
                </primary>
        </source>
</configuration>

Resolution

The standard design of the SMARTS event listener is to close and re-establish the connection to the SMARTS broker after the <idle-timeout> elapse. This process checks the broker at the defined interval and ensures that an active connection to the broker is maintained.  This avoids the scenario of the SMARTS listener and broker using a stale connection and failing to process events. Even though the message has been classified by the log handler as a WARNING, in this scenario, it is by design and is not cause for concern.