Scheduled reports failing with "Caused by: java.lang.OutOfMemoryError: Java heap space" error

Products

• VMware Smart Assurance

Issue/Introduction

Symptoms:

Certain large scheduled reports don't complete, but only sometimes. Large reports will be impacted from completing.

Environment

Watch4Net/M&R - 7.x

Cause

The following entry in the task scheduler logs points to the problem:

WARNING: An error occured when accessing Database.
190404 07:43:36 : com.watch4net.apg.v2.gui.database.DatabaseException: An error occurred when accessing the Database APG-DB.
190404 07:43:36 :     at com.watch4net.apg.v2.gui.database.global.GlobalDatabaseAccessorTask$1.onTaskFailed(GlobalDatabaseAccessorTask.java:71)
190404 07:43:36 :     at com.watch4net.async6.EvaluatorTask.notifyComplete(EvaluatorTask.java:248)
190404 07:43:36 :     at com.watch4net.async6.EvaluatorTask.run(EvaluatorTask.java:162)
190404 07:43:36 :     at com.watch4net.apg.v2.gui.database.global.GlobalDatabaseAccessorTask.compute(GlobalDatabaseAccessorTask.java:193)
190404 07:43:36 :     at com.watch4net.async6.EvaluatorTask.computeWithMDC(EvaluatorTask.java:176)
190404 07:43:36 :     at com.watch4net.async6.EvaluatorTask.run(EvaluatorTask.java:158)
190404 07:43:36 :     at com.watch4net.apg.v2.gui.database.DefaultGlobalDatabaseAccesssor.run(DefaultGlobalDatabaseAccesssor.java:229)
190404 07:43:36 :     at com.watch4net.apg.v2.gui.database.DefaultGlobalDatabaseAccesssor.getAggregatedGraphData(DefaultGlobalDatabaseAccesssor.java:97)
190404 07:43:36 :     at com.watch4net.apg.v2.gui.database.ParallelGlobalDatabaseAccesssor.getAggregatedGraphData(ParallelGlobalDatabaseAccesssor.java:184)
190404 07:43:36 :     at com.watch4net.apg.v2.gui.database.StaticDatabaseAccessor.getAggregatedGraphData(StaticDatabaseAccessor.java:235)
190404 07:43:36 :     at com.watch4net.apg.v2.gui.formula.parameter.definition.FilterFormulaParameterDefinition.feedFormula(FilterFormulaParameterDefinition.java:148)
190404 07:43:36 :     at com.watch4net.apg.v2.gui.formula.evaluation.FeedFormulaParameterTask.compute(FeedFormulaParameterTask.java:56)
190404 07:43:36 :     at com.watch4net.async6.EvaluatorTask.computeWithMDC(EvaluatorTask.java:176)
190404 07:43:36 :     at com.watch4net.async6.EvaluatorTask.run(EvaluatorTask.java:158)
190404 07:43:36 :     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
190404 07:43:36 :     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
190404 07:43:36 :     at java.lang.Thread.run(Thread.java:745)
190404 07:43:36 : Caused by: java.lang.OutOfMemoryError: Java heap space

Resolution

To resolve the issue, do the following:

1. Go to the frontend server and add following line in file /APG_HOME/Web-Applications/APG-Frontend/APG/conf/common.properties:

    scheduled.report.global.xmx=<heap_size>m

2. Make sure frontend has enough free memory left to provide to scheduled reports (Check "free -m" output for available memory)

3. Re-schedule the report and allocate more memory to allow the scheduled task to complete. 

    In this case, 4GB has been allocated to allow the report to complete.

    Additionally, you can also configure a max duration (execution timeout) of 4 hours,  this will allow the scheduled task to run for the specified period, and If it doesn’t manage to complete in 4h, it will be automatically aborted (you can change this accordingly)



4. Restart tomcat service on frontend with the following command from /APG_HOME/bin:
 ./manage-modules.sh service restart tomcat Default

Workaround:
There are no other workaround to resolve this issue.