Error: SERVER_RESTART_REQUIRED alert and StackOverflowError in DeepReferenceBuilder

Products

• Automation Analytics & Intelligence

Issue/Introduction

A critical SERVER_RESTART_REQUIRED alert appears stating that scheduler updates partially completed and the server must be restarted to get new events ․​​​​​‌​‍

 

 

ERROR MESSAGE: "Scheduler updates partially completed; server must be restarted to get new events" "java․lang․StackOverflowError at com․termalabs․server․helpers․DeepReferenceBuilder․buildReferencedJobSet"

 

 

SYMPTOMS:

• Alert Type: SERVER_RESTART_REQUIRED

   

• Alert Severity: critical

   

• The server is in a state where only a full restart is considered safe to pick up new events consistently

Cause

A StackOverflowError occurs during the POST_UPDATE phase due to unbounded mutual recursion in DeepReferenceBuilder while walking job references for exceptionally large job streams․ This causes a partial commit state․ Increasing the thread stack size mitigates this limitation․

Resolution

1. Restart the AAI service so events can process again.

 

2. ENABLE DEBUG LOGGING

AAI Configuration Tool -> Logging Tab

Click "Add by name" and enter: com․termalabs․server․app․jobstream․JobStreamBuilder Set the logging level to DEBUG

 

3. Force an update of the scheduler job defintions from the Web UI by going to System Monitoring, click on the scheduler, and click Update.

 

4. The updated logging should have DEBUG JobStreamBuilder just before the stackOverflow error like:

DEBUG [JobStreamBuilder] Rebuild jobs for jobstream <jobstreamName>

ERROR [ConverterServiceControl] Error occurred after partially committing updates for scheduler <schedulerName>

 

5. Delete the jobstream in the DEBUG logging and force an update of the scheduler definitions again and you should no longer get the error, unless another jobstream is causing the issue.

 

6. BACKUP CONFIGURATION FILE

Navigate to the root of the AAI installation directory Locate the file jaws․sh․vmoptions Create a backup copy before making edits

EXPECTED: A restorable backup is secured

 

 

7․ MODIFY jaws.sh.vmoptions 

Open jaws․sh․vmoptions for editing Locate the following stack overflow configuration section:

Uncomment the following option if run into stack overflow issue

#-Xss6m

Remove the # symbol from the parameter to uncomment it:

 

Uncomment the following option if run into stack overflow issue

-Xss6m

 

8. Restart AAI for the change to take affect

Add the problem job stream back to the system and verify there is no error after the next job definition download