Seeing a Stats Aggregation Out of Memory (OOM) error and the EM shuts down.
Article ID: 6205
Updated On:
Products
Issue/Introduction
Seeing the Introscope Enterprise Manager shutting down with the following error:
[ERROR] [Thread-66] [Manager.com.timestock.tess.services.processors.StatsAggregator] [CheckDatabase.Thread1] [com.timestock.tess.services.service.CheckDatabase$CheckDatabasePeriodically] Unexpected error checking database
java.lang.OutOfMemoryError: Java heap space
Environment
Cause
Insufficient heap is assigned to APM CE (CEM) Stats Aggregation Service.
Resolution
Heap Tuning for Stats Aggregation Service
To tune the JVM heap settings for the daily statistics aggregation process, edit the dailystats.jvmArgs property in the <EM_Home>/config/tess-default.properties file (To not have this value overridden during an upgrade, use tess-customer.properties instead.)
To overcome this issue we need to increase heap for daily aggregation. Before doing so, make sure that this machine enough physical RAM.
As an example: EM (6GB) + Aggregation (4GB) + extra (2GB) = 12GB.
Backup the tess-default.properties or tess-customer.properties file on the Introscope Enterprise Manager where CEM Stats Aggregation Service runs.
To increase Java heap for daily aggregation, open the tess-default.properties or tess-customer.properties file and increase the heap size as shown below.
As an example: From Default 256MB to 4GB of minimum and maximum settings.
dailystats.jvmArgs= -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+UseBiasedLocking -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -Dlog4j.configuration=log4j.properties -Xms256M -XX:ThreadStackSize=128
with this
dailystats.jvmArgs= -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -XX:+UseBiasedLocking -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -Dlog4j.configuration=log4j.properties -Xms4096m -Xmx4096m -XX:ThreadStackSize=128
Then restart the Introscope Enterprise Manager running the APM CE (CEM) Stats Aggregation Service.
Feedback
