RMI Log error "getBacklogAndHistory : java.lang.OutOfMemoryError: Java heap space"

book

Article ID: 209540

calendar_today

Updated On:

Products

CA Automic Applications Manager (AM)

Issue/Introduction

The below errors are found in the RMI Server log. Logging into the client may throw a similar RMI error.

ErrorMsg: AwE-5128 Client Request Error
Details: getBacklogAndHistory
Agent error : 192.168.1.164:1099:getBacklogAndHistory : java.lang.OutOfMemoryError: Java heap space
 at com.appworx.server.data.D$_A.doRun(RemoteClientSocketManager.java:181)
 at com.uc4.be.threading.AbstractWorker.run(AbstractWorker.java:367)
 at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
 at java.util.Arrays.copyOf(Arrays.java:3236)
 at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
 at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
 at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
 at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1842)
 at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1695)
 at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
 at java.math.BigInteger.writeObject(BigInteger.java:4589)
 at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1155)
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
 at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
 at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
 at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
 at java.math.BigDecimal.writeObject(BigDecimal.java:3800)
 at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1155)
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
 at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
 at java.util.ArrayList.writeObject(ArrayList.java:768)
 at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1155)
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
java.lang.OutOfMemoryError: Java heap space
 at java.util.Arrays.copyOf(Arrays.java:3236)
 at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
 at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
 at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
 at java.io.ObjectOutputStream$BlockDataOutputStream.write(ObjectOutputStream.java:1842)
 at java.io.ObjectOutputStream$PutFieldImpl.writeFields(ObjectOutputStream.java:1695)
 at java.io.ObjectOutputStream.writeFields(ObjectOutputStream.java:482)
 at java.math.BigInteger.writeObject(BigInteger.java:4589)
 at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1155)
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
 at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
 at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
 at java.io.ObjectOutputStream.defaultWriteObject(ObjectOutputStream.java:441)
 at java.math.BigDecimal.writeObject(BigDecimal.java:3800)
 at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1155)
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
 at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
 at java.util.ArrayList.writeObject(ArrayList.java:768)
 at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1155)
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)

Cause

One of the main reasons this error occurs is if the Backlog and History has grown exponentially. This could happen in an environment that has not been used in a long time allowing the Backlog and History to grow into the 1000's or 10,000's.

Environment

Release : 9.3

Component : APPLICATIONS MANAGER

Resolution

It is recommended that Support is contacted to confirm issue and a back up of the database is taken before running the below statements as the AM Oracle User:

SQL> delete from so_job_queue; 

SQL> truncate table aw_job_queue_activity; 

SQL> update so_job_history set so_status=32, so_status_name='FINISHED' where so_status_name in ('INITIATED','RUNNING','STAGED','STAGED_PW','STARTED','STARTING','QUEUE WAIT','PRED WT HOLD','PRED WAIT','KILLING','DATE PENDING','CONDITN WAIT','AGENT WAIT')

A commit should be done between each statement. 

Powered by Wolken Software