AAI Schedulers showing 'Behind' status due to StackOverflowError in DeepReferenceBuilder

Products

• Automation Analytics & Intelligence

Issue/Introduction

Users may observe the following symptoms in Automation Analytics & Intelligence (AAI):

• Schedulers consistently show a "Behind" status in the AAI Web UI.
• Execution data and job definitions are not synchronized properly.
• Restarting AAI services provides only a temporary workaround; the issue returns during subsequent converter runs.

Error Messages

The jaws.log file reveals critical errors during the POST_UPDATE phase:

ERROR [SchedulerHelper] Error while committing updates for scheduler "[SchedulerName]"com.termalabs.server.domain.api.SchedulerUpdate$UpdateFailedException: SchedulerUpdate failed in the POST_UPDATE phase...Caused by: java.lang.StackOverflowError    at com.termalabs.server.helpers.DeepReferenceBuilder.buildReferencedJobSet (recursive)    at com.termalabs.server.helpers.DeepReferenceBuilder.getReferencedJobSet (recursive)

A corresponding warning alert may also appear: WARN [SchedulerHelper] Scheduler updates partially completed; server must be restarted to get new events.

Cause

The issue is caused by a StackOverflowError occurring during the recursive traversal of complex job dependency graphs or exceptionally large job streams.

While users often increase the Java Heap Memory (-Xmx), this does not resolve the issue because the Thread Stack Size (-Xss) is separate. When the dependency graph depth exceeds the default stack size, the JVM thread exhausted its allocated stack space, leading to a partial commit state in the database.

Resolution

To resolve this issue, the JVM thread stack size must be increased to allow for deeper recursion during the reference building process.

Step 1: Modify JVM Options

1. Navigate to the AAI installation root directory.
2. Locate the following two configuration files:
   • Automation Analytics & Intelligence (application).exe.vmoptions (Windows) or jaws.sh.vmoptions (Linux)
   • Automation Analytics & Intelligence (service).exe.vmoptions (Windows)
3. Open the files in a text editor.
4. Locate the line containing -Xss (it may be commented out as #-Xss6m).
5. Uncomment or add the parameter to set the stack size to 6MB:
    

   -Xss6m

6. Save the changes to both files.

Step 2: Restart AAI Services

1. Stop the AAI services/application.
2. Start the AAI services to allow the JVM to initialize with the new stack settings.

Step 3: Verification

1. Log in to the AAI Web UI.
2. Monitor the Schedulers under System Monitoring.
3. Verify that the schedulers complete their update cycle and move from "Behind" to "On Time".
4. Review the jaws.log to ensure no further StackOverflowError messages are generated.

Additional Information

Additional Troubleshooting

If the error persists even after increasing the stack size to 6MB, it may indicate a circular dependency or an atypically large job stream (~25,000+ jobs). In such cases:

1. Enable DEBUG logging for com.termalabs.server.app.jobstream.JobStreamBuilder to identify the specific job stream causing the failure.
2. Consider implementing Auto-Trim or splitting the pathological job stream into smaller, more manageable units.