EM orderly shutdown due to Stack Overflow

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• INTROSCOPE

Issue/Introduction

Description:

In certain situations, which has more commonly been found during an upgrade of EM from an older release to newer 9.1.x release, the EM would start up but would soon trigger an orderly shutdown due to stack overflow, with following error seen in the IntroscopeEnterpriseManager.log:

[ERROR] [ServicesCheck.Thread1] [Manager] Uncaught Exception in Enterprise Manager:  In thread ServicesCheck.Thread1 and the message is java.lang.StackOverflowError
[VERBOSE] [ServicesCheck.Thread1] [Manager] Uncaught Exception in thread ServicesCheck.Thread1
java.lang.StackOverflowError
at org.hibernate.hql.ast.util.NodeTraverser.visitDepthFirst(NodeTraverser.java:41)
at org.hibernate.hql.ast.util.NodeTraverser.visitDepthFirst(NodeTraverser.java:42)
at org.hibernate.hql.ast.util.NodeTraverser.visitDepthFirst(NodeTraverser.java:42)
...

Solution:

Lowering the rows fetched by a query will prevent overflowing EM stacks.

The impact will be that lesser rows will be fetched per query and it may require more time/query cycle to fetch more rows. However the change comes with the more significant benefit of preventing the EM from crashing.

To lower the rows fetched per query, please add the following entry to the EM_HOME/config/tess-customer.properties file. If this file does not already exist create it according to the instructions for modifying the tess-default.properties file in the Configuration and Administration Guide:

Change:
db.rowsFetchedPerQuery=10000
to:
db.rowsFetchedPerQuery=5000

Then restart the EM. If 5000 is not low enough, try 1000.
This issue has been addressed in APM release 9.1.4.0 and onwards, by reducing the default value for the db.rowsFetchedPerQuery property to 5000 in the tess-default.properties file.

Environment

Release:
Component: APMISP