Long delays when starting Weblogic application server with APM 9.x Java Agent enabled.

Products

• APP PERF MANAGEMENT
• CA Application Performance Management Agent (APM / Wily / Introscope)
• CUSTOMER EXPERIENCE MANAGER
• INTROSCOPE

Issue/Introduction

A noticeable delay is observed when starting up Weblogic application server with 9.x version of the APM Java agent. (This can also occur in other application servers such as Websphere.)

Cause

Seeing error in the agent logs:

<Error> <WebLogicServer> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1368492926910> <BEA-000337> <[STUCK] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "648" seconds working on the request "[email protected]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.

The resulting stack trace will show signs of Introscope deep inheritance classes:

com.wily.diagnos.personality.java.classfile.DGClass.isInstanceOf(DGClass.java:1134)
com.wily.introscope.agent.db.DeepInheritanceHelper.addInheritanceMatchRequest(DeepInheritanceHelper.java:139)
com.wily.diagnos.cmp.classmatcher.DGDeepInheritedClassMatcher.executeMatches(DGDeepInheritedClassMatcher.java:60)
com.wily.diagnos.cmp.classmatcher.DGDeepInheritedClassMatcher.IClassMatcher_matches(DGDeepInheritedClassMatcher.java:110)

Environment

Java Agent 9.6x and 9.7x.

Resolution

By disabling deep inheritance in IntroscopeAgent.profile, this will resolve the issue but instrumentation of classes that are "remotely" inherited from a Class/Interface defined in pbd won´t happen. (See the APM Java Agent Guide for more information about deep inheritance). By disabling the property, only 2 levels of class inheritance are checked

introscope.autoprobe.deepinheritance.enabled=false

Any classes that are missing a result of the change can be traced explicitly with normal pbd statements.