AAI Response Time is slow and Jammer process not running and logs indicate a possible memory issue.

book

Article ID: 197855

calendar_today

Updated On:

Products

Automic Automation Intelligence

Issue/Introduction

AAI Response time is slow and in looking at the jaws.log I do not seen any entries for Jammer running.  I see a lot of MemoryLogger, QueryTimingService and periodic Memory Status messages.  

 

Deadlock messages are seeing in the logs and the the memory utilization consumption is very high. 

INFO [MemoryLogger] Memory status: 89.4% (3.18 GB of 3.56 GB) in use; 3.41 GB allocated; high water: 3.18 GB (6.0.1-2)

 

WARN [ThreadPoolAsynchronousRunner] com[email protected]161ec109 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!

 

WARN [ThreadPoolAsynchronousRunner] com[email protected]161ec109 -- APPARENT DEADLOCK!!! Complete Status: 
 Managed Threads: 10
 Active Threads: 0
 Active Tasks: 
 Pending Tasks: 
  com.mchange.[email protected]6285edec
Pool thread stack traces:
 Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,ServerService ThreadGroup]
  java.lang.Object.wait(Native Method)
  com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
 Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#5,5,ServerService ThreadGroup]
  java.lang.Object.wait(Native Method)
  com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
 Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#7,5,ServerService ThreadGroup]
  java.lang.Object.wait(Native Method)
  com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
 Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,ServerService ThreadGroup]
  java.lang.Object.wait(Native Method)
  com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
 Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#8,5,ServerService ThreadGroup]
  java.lang.Object.wait(Native Method)
  com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
 Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#4,5,ServerService ThreadGroup]
  java.lang.Object.wait(Native Method)
  com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
 Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#3,5,ServerService ThreadGroup]
  java.lang.Object.wait(Native Method)
  com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
 Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#6,5,ServerService ThreadGroup]
  java.lang.Object.wait(Native Method)
  com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
 Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,ServerService ThreadGroup]
  java.lang.Object.wait(Native Method)
  com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)
 Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#9,5,ServerService ThreadGroup]
  java.lang.Object.wait(Native Method)
  com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:534)

 

Environment

Release : 6.0.1

Component : AUTOMIC AUTOMATION INTELLIGENCE ENGINE

Resolution

Increase the MIN RAM and MAX RAM in the AAI Configuration tool.

1. The configuration tool can be found in the /<install dir>/config/ directory and can be launched by running "run.sh" on Linux or "run.bat" on Windows.

         **Note** For Linux you must have x11 forwarding on.

2. Go to the "Other" tab on the Config tool and update the MIN and MAX RAM. 

    Typically you should set the MAX RAM to 2-3x the max value you see being used in the jaws.log and the MIN RAM can be slightly lower.

3. After updating the RAM, restart the AAI service.

4. Verify the RAM utilization changes in the Jaws log by looking for a line like below:

INFO [MemoryLogger] Memory status: 32.4% (2.88 GB of 8.89 GB) in use; 8.89 GB allocated; high water: 3.99 GB (6.0.1-2)

5. After waiting a few minutes verify that you see the Jammer process running in the logs by looking for a line like "INFO [JammerManager] Started to run JammerManager sequence"

 

The problem with the Jammer not running properly was due to lack of memory, and was causing deadlock errors once RAM was increased to Min 8GB and Max 10GB, the Jammer started running properly.

The memory settings were not updating originally due to a defect we provided a work around for documented in case 32176996.

Additional Information

If you are running Windows and AAI 6.0.x or later you may need to follow the work around in the KB bellow to update the Memory settings.

https://knowledge.broadcom.com/external/article?articleId=196453

Attachments