DDM sync issue to Spectrum Report Manager database - Expection occurs while retrieving events for <LANDSCAPE>

book

Article ID: 212076

calendar_today

Updated On:

Products

CA Spectrum

Issue/Introduction

SRM Synchronization for the some landscapes is not taking place.  From the Spectrum WEB GUI -> Administration -> SPECTRUM Status, the Last Event Time for some landscapes is stuck to several days before.

The following exception is generated repeatedly in the Spectrum Tomcat log file:

Aug 16, 2021 19:14:17.601 (SRM/EventPoller/<LANDSCAPE>) (SRM_Events) - (ERROR) - Expection occurs while retrieving events for <LANDSCAPE>. retrying in cycle 

 

Caused by: exception com.aprisma.spectrum.core.idl.CsCException.CsCSpectrumException {

com.aprisma.spectrum.core.idl.CsCError.CsCError_e error=DOES_NOT_EXIST,

java.lang.String messageFormatID="CsCEventDomain.getNextEventListByAttrFilter.requestID.DOES_NOT_EXIST",

com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue[] messageValueList={union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="CsEventDSrvc.cc"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="%I% %G% %U%"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

int intValue=962

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="<LANDSCAPE>"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

int intValue=0

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="<LANDSCAPE>/CsCEventDomain"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

int intValue=1843

}}

}

        at com.aprisma.spectrum.core.idl.CsCException.CsCSpectrumExceptionHelper.read(CsCSpectrumExceptionHelper.java:40)

        at com.aprisma.spectrum.core.idl._CsCEventDomainStub.getNextEventListByAttrFilter(_CsCEventDomainStub.java:1458)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventDataModel.getEvents(EventDataModel.java:452)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventPoller.doHandleEvents(EventPoller.java:910)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventPoller.handleEvents(EventPoller.java:801)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventPollerTask.run(EventPollerTask.java:100)

        at com.aprisma.util.TimerThread.mainLoop(TimerEx.java:617)

        at com.aprisma.util.TimerThread.run(TimerEx.java:535)

 

Even after applying changes advised in KB : Event sync between Archive Manager & SRM very slow with CORBA "DOES_NOT_EXIST" exception found in Spectrum Tomcat log however the SRM event synchronization remains extremely slow.

Environment

Release : 20.2

Component : Spectrum Core / SpectroSERVER

Resolution

This is generally indicative of performance issues with regards to SRM on the OneClick server. The following are some techniques that may alleviate the issue:

  1. Under Spectrum WEB GUI->Report manager->Preferences, change the instant_event_poller_processing_interval from 5 minutes to 60 minutes to allow for greater time to process the current sync before attempting another.

  2. Change the MaxLandscapeThreads parameter from the default value 2 to the number of your landscapes in the file:

    $SPECROOT/tomcat/webapps/spectrum/WEB-INF/web.xml 

    For example, if you have 6 landscapes, then it would be:

    <context-param xmlns="">

        <param-name>com.aprisma.reportmanager.MaxLandscapeThreads</param-name>

        <param-value>6</param-value>

        <description>

          The maximum number of landscapes that will be read in parallel.

        </description>

    </context-param>



  3. Restart the Spectrum tomcat

  4. If there are no significant improvements in performance, analyze the top 10 events directly in the Archive Manager DDMdb (MySQL):

    mysql> SELECT HEX(type), COUNT(*) AS cnt FROM event GROUP BY type ORDER BY cnt DESC LIMIT 10;
    +-----------+-----------+
    | HEX(type) | cnt       |
    +-----------+-----------+
    | 10802     | 429879784 |
    | 210D88    | 111465994 |

    ...

     


    The output will show if you have any events types that are high in number, with the count in the millions. Check what events these are under OneClick -> Tools menu -> Utilities -> Event Configuration

  5. If there are events associated with unmanaged traps and possibly a trap storm whereby an un-modelled device is flooding Spectrum with traps, then the associated events can be disabled as per the following KB:

    KB : How can unmanaged traps in SPECTRUM be disabled (event 0x10802)?

  6. You can also delete the event from the Archive Manager DDMdb if it is unwanted. For example: 

    DELETE FROM event WHERE type=<EVENT_TYPE>;

  7. Then run an optimize of the DDMdb by executing the script:

    $SPECROOT/SS/DDM/db_optimize.pl

Additional Information

If the number  of events to delete (select count(*) from event where type=0x10802;)  is very huge, you could delete only a subset specifying a time range of one week at time.

eg.

delete from event where type=0x10802 and from_unixtime(utime) > '2020-05-01 00:00:00' and from_unixtime(utime) < '2020-05-07 00:00:00';

delete from event where type=0x10802 and from_unixtime(utime) > '2020-05-08 00:00:00' and from_unixtime(utime) < '2020-05-15 00:00:00';
...