Secondary Rmi Server consumes CPU/Memory resulting in a Java heap space error

Products

• CA Automic Applications Manager (AM)

Issue/Introduction

This issue can occur when running a  Secondary RMI Server for Failover. When running a secondary RmiServer (primary still running), it runs for several minutes to hours but eventually hits the below errors.

This then leads to consuming high amounts of Java memory and the RMI process failing with a Java Heap space error on the secondary RmiServer process.

The Following errors can be seen in the seondary RmiServer logs...

11:24:02.281 rfp0: .JobsData: history job was not in backlog 10796
11:24:02.281 rfp0: AwE-9999
11:24:02.281 rfp0: .AxOptions: NoErrorMsgProperties=false
ErrorMsg: AwE-9999 Internal error (11/8/22 11:24 AM)
Details: Job 10797 has a parent jobid of 10796 however
could not be found in memory. lookup object was a map false

11:24:02.281 rfp0: .LocalPredTesterImpl: Jobid 10797 came in out of order, parent jobid is 10796
11:24:02.281 rfp0: AwE-9999
ErrorMsg: AwE-9999 Internal error (11/8/22 11:24 AM)
Details: Job 10798 has a parent jobid of 10796 however
could not be found in memory. lookup object was a map false
java.lang.RuntimeException: No parent in memory error


11:24:13.328 rf: .DBAccess: getFunctionData() sID-317416 aw_web_api.aw_inc_update
11:24:13.328 rfp0: .JobsData: refreshData predTester (sync)
11:24:13.328 rf: .A: initStatement()
11:24:13.328 rfp0: .ThreadPool: suspend ptadd
11:24:13.328 rf: aw_web_api.aw_inc_update 0 maxSeq: IN:NUMERIC:java.math.BigDecimal:34325
11:24:13.328 rfp0: AwE-9999
ErrorMsg: AwE-9999 Internal error (11/8/22 11:24 AM)
Details: ptadd 6
java.lang.IllegalStateException: ThreadPool.suspend


12:08:22.127 rfp0: .LocalPredTesterImpl: Jobid 10801 came in out of order, parent jobid is 10796
12:08:22.127 rfp0: .LocalPredTesterImpl: Jobid 10802 came in out of order, parent jobid is 10796
12:08:22.127 rfp0: .LocalPredTesterImpl: Jobid 10803 came in out of order, parent jobid is 10796

12:08:26.843 rfp0: AwE-9999
12:08:26.859 rfp0: .AxOptions: NoErrorMsgProperties=false
ErrorMsg: AwE-9999 Internal error (11/8/22 12:08 PM)
Details: null
java.lang.OutOfMemoryError: Java heap space

Environment

Applications Manager 9.4.x, 9.5 to 9.5.2

Cause

This appears to be caused by having a process flow with a before request job condition = cancel process flow scheduled at a high frequency when using a secondary RmiServer.

Resolution

This is fixed in Applications Manager version 9.5.3 and 9.6

Workaround:

Disable the process flow with high frequency cancel process flow condition or run the master without the secondary Rmi server in place.