Seeing a Stats Aggregation Out of Memory (OOM) error and the EM shuts down.

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

  Seeing the Introscope Enterprise Manager shutting down with the following error:

[ERROR] [Thread-66] [Manager.com.timestock.tess.services.processors.StatsAggregator] [CheckDatabase.Thread1] [com.timestock.tess.services.service.CheckDatabase$CheckDatabasePeriodically] Unexpected error checking database
java.lang.OutOfMemoryError: Java heap space

 

 

Cause

  Insufficient heap is assigned to APM CE (CEM) Stats Aggregation Service. 

Environment

APM 10.x APM Database: Oracle (All supported releases)

Resolution

 

 Heap Tuning for Stats Aggregation Service

 To tune the JVM heap settings for the daily statistics aggregation process, edit the dailystats.jvmArgs property in the <EM_Home>/config/tess-default.properties file (To not have this value overridden during an upgrade, use tess-customer.properties instead.)

 To overcome this issue we need to increase heap for daily aggregation. Before doing so, make sure that this machine enough physical RAM.

As an example: EM (6GB) + Aggregation (4GB) + extra (2GB) = 12GB.

  Backup the tess-default.properties or tess-customer.properties file on the Introscope Enterprise Manager where CEM Stats Aggregation Service runs.

  To increase Java heap for daily aggregation, open the tess-default.properties or tess-customer.properties file  and increase the heap size as shown below.

 

As an example: From Default 256MB to 4GB of minimum and maximum settings.

dailystats.jvmArgs= -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+UseBiasedLocking -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -Dlog4j.configuration=log4j.properties -Xms256M -XX:ThreadStackSize=128

with this

dailystats.jvmArgs= -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+UseBiasedLocking -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -Dlog4j.configuration=log4j.properties -Xms4096m -Xmx4096m -XX:ThreadStackSize=128 
   
   Then restart the Introscope Enterprise Manager running the APM CE (CEM) Stats Aggregation Service.