OneClick servers are not accessible

Products

• Spectrum
• Network Observability

Issue/Introduction

We are facing an issue with all of our OneClick servers where they seems to be unreachable.

extract from catalina:

Jul 03, 2020 11:43:22.461 - Error watching attributes for models in domain 0xf00000 : java.lang.OutOfMemoryError: GC overhead limit exceeded Jul 03, 2020 11:43:27.983 (PoolThread-178: GlobalPool => BackEndEventDataModel.contentsChanged) (SLM_STAT_EVENT) - slm to begin processing historical outages for domain=0x600000 Jul 03, 2020 11:43:44.056 - Error watching attributes for models in domain 0xf00000 : java.lang.OutOfMemoryError: GC overhead limit exceeded Jul 03, 2020 11:43:47.585 (PoolThread-178: HistoricalUpdateJobPool => BackEndEventDataModel.HistoricalUpdateJob) (SLM_STAT_EVENT) - slm finished processing historical outages for domain = 0x600000 Jul 03, 2020 11:44:06.387 - Error watching attributes for models in domain 0xf00000 : java.lang.OutOfMemoryError: GC overhead limit exceeded Jul 03, 2020 11:44:28.920 - ModelRepository error: java.lang.OutOfMemoryError: GC overhead limit exceeded

Environment

Release : 10.2

Component : Spectrum Core / SpectroSERVER

Cause

Global collections were set to update in real time.

 

Resolution

All OC servers have only 8 GB of ram with 3 dedicated to tomcat. I have recommended to request more memory as we now recommend 8 for tomcat and 16 on the server.  As this will take some time to implement, we worked around it as follows:

On one OC server the tomcat memory was increased from 3 to 5 GB from catalina.sh and we could start a console on this.

We changed 2 GC from dynamic update to hourly and watched tomcat memory usage go down from 80% to much lower.

The other tomcats still only had 3 GB but we could start a console from them after the changed Global collections were reflected to the SpectroSERVER.