DDM sync issue to Spectrum Report Manager database - Exception occurs while retrieving events for <LANDSCAPE>

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

SRM Synchronization for the some landscapes is not taking place.  From the Spectrum WEB GUI -> Administration -> SPECTRUM Status, the Last Event Time for some landscapes is stuck to several days before.

The following exception is generated repeatedly in the Spectrum Tomcat log file:

Aug 16, 2021 19:14:17.601 (SRM/EventPoller/<LANDSCAPE>) (SRM_Events) - (ERROR) - Expection occurs while retrieving events for <LANDSCAPE>. retrying in cycle 

 

Caused by: exception com.aprisma.spectrum.core.idl.CsCException.CsCSpectrumException {

com.aprisma.spectrum.core.idl.CsCError.CsCError_e error=DOES_NOT_EXIST,

java.lang.String messageFormatID="CsCEventDomain.getNextEventListByAttrFilter.requestID.DOES_NOT_EXIST",

com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue[] messageValueList={union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="CsEventDSrvc.cc"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="%I% %G% %U%"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

int intValue=962

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="<LANDSCAPE>"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

int intValue=0

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="<LANDSCAPE>/CsCEventDomain"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

int intValue=1843

}}

}

        at com.aprisma.spectrum.core.idl.CsCException.CsCSpectrumExceptionHelper.read(CsCSpectrumExceptionHelper.java:40)

        at com.aprisma.spectrum.core.idl._CsCEventDomainStub.getNextEventListByAttrFilter(_CsCEventDomainStub.java:1458)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventDataModel.getEvents(EventDataModel.java:452)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventPoller.doHandleEvents(EventPoller.java:910)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventPoller.handleEvents(EventPoller.java:801)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventPollerTask.run(EventPollerTask.java:100)

        at com.aprisma.util.TimerThread.mainLoop(TimerEx.java:617)

        at com.aprisma.util.TimerThread.run(TimerEx.java:535)

 

Even after applying changes advised in KB : Event sync between Archive Manager & SRM very slow with CORBA "DOES_NOT_EXIST" exception found in Spectrum Tomcat log however the SRM event synchronization remains extremely slow.

Environment

Release : 20.2,21.2

Component : Spectrum Core / SpectroSERVER

Resolution

This is generally indicative of performance issues with regard to SRM on the OneClick server. The following are some techniques that may alleviate the issue:

1. Under Spectrum WEB GUI->Report manager->Preferences, change the instant_event_poller_processing_interval from 5 minutes to 60 minutes to allow for greater time to process the current sync before attempting another.
   
   
2. Change the MaxLandscapeThreads parameter from the default value 2 to the number of your landscapes in the file:
   
   

   $SPECROOT/tomcat/webapps/spectrum/WEB-INF/web.xml 
   

   For example, if you have 6 landscapes, then it would be:
   
   

   <context-param xmlns="">

       <param-name>com.aprisma.reportmanager.MaxLandscapeThreads</param-name>

       <param-value>6</param-value>

       <description>

         The maximum number of landscapes that will be read in parallel.

       </description>

   </context-param>

   
   
   
3. Restart the Spectrum tomcat
   
   
4. If there are no significant improvements in performance, analyze the top 10 events directly in the Archive Manager DDMdb (MySQL):
   
   

   mysql> SELECT HEX(type), COUNT(*) AS cnt FROM event GROUP BY type ORDER BY cnt DESC LIMIT 10;
   +-----------+-----------+
   | HEX(type) | cnt       |
   +-----------+-----------+
   | 10802     | 429879784 |
   | 210D88    | 111465994 |

   ...

    

   
   The output will show if you have any events types that are high in number, with the count in the millions. Check what events these are under OneClick -> Tools menu -> Utilities -> Event Configuration
   
   
5. If there are events associated with unmanaged traps and possibly a trap storm whereby an un-modeled device is flooding Spectrum with traps, then the associated events can be disabled as per the following KB:
   
   KB : How can unmanaged traps in SPECTRUM be disabled (event 0x10802)?
   
   
6. You can also delete the event from the Archive Manager DDMdb if it is unwanted. For example: 
   
   

   DELETE FROM event WHERE type=<EVENT_TYPE>;
   
   

7. Then run an optimize of the DDMdb by executing the script (if you want to reclaim unused space - marked as deleted to be reused):
   
   

   $SPECROOT/SS/DDM/db_optimize.pl
   
   

8. Instead of deleting the events from the DDMdb (step 6), you can create an Event Processing Filter on the SRM machine.
   

   a. Copy the event-processing-filter.xml and event-processing-filter-schema.xsd files to the 'custom' directory. For example, see the following syntax:

   cp <$SPECROOT>/tomcat/webapps/spectrum/WEB-INF/repmgr/config/event-processing-filter-schema.xsd  <$SPECROOT>/custom/repmgr/config/

   cp <$SPECROOT>/tomcat/webapps/spectrum/WEB-INF/repmgr/config/events/event-processing-filter.xml  <$SPECROOT>/custom/repmgr/config/

   b. Edit the event-processing-filter.xml to reflect your selected filtering strategy.

   For example, see the following syntax:

   <ignore>

   <event-type>0x10802</event-type>

   <model>0x00d40010</model>

   </ignore>

   Note: You can only ignore events that are associated with specific models or event types.

   c. Restart the Tomcat service.

   The specified event processing filters are now in effect.

   Note: Events code listed in Appendix A. DX NetOps Spectrum Events Used by Report Manager (broadcom.com) should not be added in the event-processing-filter.xml file.

Additional Information

If the number  of events to delete (select count(*) from event where type=0x10802;)  is very huge, you could delete only a subset specifying a time range of one week at time.

eg.

delete from event where type=0x10802 and from_unixtime(utime) > '2020-05-01 00:00:00' and from_unixtime(utime) < '2020-05-07 00:00:00';

delete from event where type=0x10802 and from_unixtime(utime) > '2020-05-08 00:00:00' and from_unixtime(utime) < '2020-05-15 00:00:00';
...