Spectrum SRM synchronization falling behind by a day or more

Products

• CA Spectrum
• DX NetOps

Issue/Introduction

While synchronization is occurring, it is never up to date with the current date as shown under;

 

Spectrum WEB GUI -> Administration -> Report Manager -> Spectrum Status -> Last Event Time​

After increasing the Lock Timeout wait time by adding the following line to the $SPECROOT/mysql/my-spectrum.cnf file:

innodb_lock_wait_timeout = 600

Then restarting MySQL and the OC Tomcat server, but still the events do not sync up to date. The following error is repeatedly observed in the OneClick server log:

Nov 23, 2021 06:20:38.054 (SRM/EventPoller/spectro-server) (SRM_Events) - (ERROR) - Expection occurs while retrieving events for ca-spectro-server. retrying in cycle 

Caused by: exception com.aprisma.spectrum.core.idl.CsCException.CsCSpectrumException {

com.aprisma.spectrum.core.idl.CsCError.CsCError_e error=DOES_NOT_EXIST,

java.lang.String messageFormatID="CsCEventDomain.getNextEventListByAttrFilter.requestID.DOES_NOT_EXIST",

com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue[] messageValueList={union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="CsEventDSrvc.cc"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="%I% %G% %U%"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

int intValue=962

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="SPECTRO-SERVER"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

int intValue=0

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

java.lang.String textString="enms-ca-spectro-server/CsCEventDomain"

},union com.aprisma.spectrum.core.idl.CsCAttribute.CsCValue {

int intValue=145

}}

}

        at com.aprisma.spectrum.core.idl.CsCException.CsCSpectrumExceptionHelper.read(CsCSpectrumExceptionHelper.java:40)

        at com.aprisma.spectrum.core.idl._CsCEventDomainStub.getNextEventListByAttrFilter(_CsCEventDomainStub.java:1458)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventDataModel.getEvents(EventDataModel.java:452)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventPoller.doHandleEvents(EventPoller.java:910)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventPoller.handleEvents(EventPoller.java:801)

        at com.aprisma.spectrum.app.repmgr.dc.event.EventPollerTask.run(EventPollerTask.java:100)

        at com.aprisma.util.TimerThread.mainLoop(TimerEx.java:617)

        at com.aprisma.util.TimerThread.run(TimerEx.java:535)

Environment

DX NetOps Spectrum 21.2, 22.2

Cause

This is usually the result of performance related issues especially when the SRM DB is very large (for example, several hundred GBs in size).

Resolution

Increasing the buffer pool memory and instances can help in these cases. The defaults are set to:

innodb_buffer_pool_size = 3072M

innodb-buffer-pool-instances = 2

Each pool instance requires ~1GB - 1.5GB of memory. So if the buffer pool size is doubled to 6GB, then the system should be able to get 4 instances since if it has 48GB of RAM for example (24GB for the buffer pools, and 24GB for everything else - OS, Spectrum Tomcat, Mysql Engine etc) . These are set in the file:

$SPECROOT/mysql/my-spectrum.cnf

You can set it to:

innodb_buffer_pool_size = 6144M

innodb-buffer-pool-instances = 4

Then stop Spectrum Tomcat and processd on the OneClick server (to restart mysqld).