Spectrum Oneclick Topologies 'hang'

book

Article ID: 185925

calendar_today

Updated On:

Products

CA Spectrum CA eHealth

Issue/Introduction

We regularilly see OC performance issues, where random containers in the topology view remain 'initializing'.
When a topology stays in initializing, it seems to stay like that for the duration of the oneclick session. 
After restarting the Console or tomcat, it will happen on a different container after some time.
It also happens with WebApp beta, but to a lesser extent.

I've checked cpu/memory of the OC and SS but don't see anything really out of order.
When the problem happens, we see this in catalina.out :

WARNING: Client lost config response: -1 - reprocessing configuration update...
java.lang.NullPointerException
        at java.lang.Integer.decode(Integer.java:1161)
        at com.aprisma.spectrum.app.web.servlet.ClientPollServlet.doPostImpl(ClientPollServlet.java:3165)
        at com.aprisma.spectrum.app.web.servlet.ClientPollServlet.doPost(ClientPollServlet.java:3100)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:668)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)








Cause

This is seen on OneClick servers integrated with CA Performance Center.

Environment

Release : 10.3

Component : Spectrum Core / SpectroSERVER

Resolution

We document a minimum of eight GB of RAM for oneClick only servers.
The default Ram dedicated to Tomcat is 4GB, which is sufficient in smaller environments and leave 4 for the OS which should be more than enough with room to increase tomcat RAM.
This is the minimum but we recommend 16GB for better performance.

Tomcat memory is configured from the oneClick web Administration page " web server memory". From this page we can't see how much memory Tomcat is using which informs our decision later we need to increase it or not.

If performance issues are seen on one click servers, ram recommendations increase depending on the numbers seen on the aforementioned page. Performance issues are regularly seen in oneClick servers in busy environments or those that have Integrations with other products as CAPC UIM; and VNA etc. in these situations we will need to increase the server memory the memory dedicated to Tomcat as a percentage of that.

For performance issues seen on consoles, we recommend increasing the client memory to 1 GB from the current 750 megabytes on the web admin page.
We also recommend increasing the ModelRepository.ThrottledModelReadCount to 7,500 from the current 5000 in the <SPECROOT>/tomcat/webapps/spectrum/WEB-INF/web.xml.