Error messages cause java memory consumption until core dump

Products

• CA Automic Applications Manager (AM)

Issue/Introduction

Error messages cause java memory consumption until core dump

Environment

Release: AAMOS499000-8.0-Automic Applications Manager-OS400 Agent
Component:

Resolution

Detailed Description and Symptoms

Redundant Error messages cause java memory consumption until core dump.

Investigation

In this example a job is going into an Aborted status. There is a condition on the job set to restart on abort.  The job is continuing to abort, however it being reset via the condition.  The jobs parent id eventually clears java memory, however the job is still continued to reset via the condition. After multiple resets (47 in example) the parent jobid is no longer available in memory for reference.  Here is an example of this error seen in the logs:

 

 03:50:00.678 rfp0: AwE-9999

 03:50:00.679 rfp0: .AxOptions: NoErrorMsgProperties=false

 ErrorMsg: AwE-9999 Internal error (9/13/11 3:50 AM)

 Details: Job 2681163.47 has a parent jobid of 2681162 however

 could not be found in memory.  lookup object was a map false

 java.lang.RuntimeException: No parent in memory error

 

 *Note the .47 appended to the jobid (2681163.47) indicates that the job was reset 47 times.

 

 This then causes the job 2681163.47 to continuously check for the parent 2681162. The continuous check for parents on multiple jobs puts a resource strain on the java vm, until it eventually dumps.

 

07:14:46.978 Timer-5: .A: initStatement()

07:14:46.979 rfp0: .AxOptions: NoErrorMsgProperties=false

ErrorMsg: AwE-9999 Internal error (9/13/11 7:14 AM)

Details: null

java.lang.OutOfMemoryError: Java heap space

at java.util.ArrayList.ensureCapacity(ArrayList.java:169)

at java.util.ArrayList.add(ArrayList.java:351)

at com.appworx.master.predtester.LocalPredTesterImpl.jobStatusChanged(LocalPredTesterImpl.java:960)

at com.appworx.master.predtester.LocalPredTesterImpl.resumePredAnalysis(LocalPredTesterImpl.java:1718)

at com.appworx.server.data.JobsData.?(JobsData.java:2501)

at com.appworx.server.data.JobsData.?(JobsData.java:2002)

at com.appworx.server.data.JobsData.A(JobsData.java:81)

at com.appworx.server.data.JobsData$_E.doRun(JobsData.java:3549)

at com.automic.be.threading.AbstractWorker.run(AbstractWorker.java:378)

at java.lang.Thread.run(Thread.java:595)

Solution

We can increase the java garbage collection allocation by modifyingthe $AW_HOME/data/inst.pl file to the following.

Original:
$START_OPTION_RMI= $ENV{START_OPTION_RMI} if ($ENV{START_OPTION_RMI});

$command = "$spawn -p $pidfile\"$java -DAGENT=$OPER $Xrs -Xmx$java_mb $START_OPTION_RMI-DAW_HOME=$AW_HOME -DOsType=$OSTYPE $rmidebugcom.appworx.server.data.AxRmiServer\" 1>$logfile 2>&1";

Modified:

$START_OPTION_RMI= $ENV{START_OPTION_RMI} if ($ENV{START_OPTION_RMI});

$command = "$spawn -p $pidfile\"$java -DAGENT=$OPER $Xrs -XX:PermSize=256m -XX:MaxPermSize=256m-Xmn1024m -Xms2048m -Xmx2048m $START_OPTION_RMI -DAW_HOME=$AW_HOME-DOsType=$OSTYPE $rmidebug com.appworx.server.data.AxRmiServer\"1>$logfile 2>&1";

Thischange will require a restart of their Applications Manager processes.