Scheduler task throws exception org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]

Products

• CA API Gateway

Issue/Introduction

Use Case:

1. There is a Scheduler Task running every second, which may take a few seconds to complete depending on the API load. 

2. The environment is using buffers (Accumulate Data in Memory Assertion) to read in the Scheduler Task and send data to a backend. 

The policies in the APIs are writing to this buffer and the Scheduler task is reading from this buffer. 

Since the Scheduler Task may take a few seconds to complete, and the frequency used is every second, there would be multiple threads for running the Scheduler task every second.

On high load, it throws

2024-01-10T08:59:54.102+0000 SEVERE  634    org.quartz.core.ErrorLogger: Job (DEFAULT.2c2f9a20a7530cc8bdc9547ae545bcf5 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
	at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at jdk.scripting.nashorn/jdk.nashorn.internal.runtime.ConsString.flatten(Unknown Source)
	at jdk.scripting.nashorn/jdk.nashorn.internal.runtime.ConsString.flattened(Unknown Source)

The memory and CPU are well within the limits of the VM. 

Is there something wrong with the Scheduler task policy configuration? Are there some limits on Scheduler Tasks and, if so, how to increase them? 

Environment

CA API Gateway 11.0

Cause

There was an issue with machine resources and policy configuration.

Resolution

Things to consider:

1. Adding any assertions to a processing request requires more CPU. The amount of required CPU depends on each specific environment. You should monitor CPU utilization while running a production-like load in a test environment.

2. Policy language is interpreted at runtime and seemingly "simple" operations may be more complex from the CPU utilization perspective than at first glance.
   
For example, "Capture some data into a variable" might imply that the message and its headers had to be loaded and parsed first before capturing some data to a variable. Also such simple policies are not representative of production performance because production policies are orders of magnitude more complex.