How to Monitor Active Hung Threads using Introscope Agent?

Products

• CA Application Performance Management (APM / Wily / Introscope)
• CA Application Performance Management Agent (APM / Wily / Introscope)

Issue/Introduction

How to Monitor Active Hung Threads using Introscope Agent?

Environment

CA Application Performance Management 10.1 awards.

Resolution

OPTION 1: Use EPAgent LogReader scripts.

a) If you are using Weblogic and Webpshere, you can use WebsphereLogReader or Weblogic LogReader. For example, for WebsphereLogReader, update the websphereLogReader.cfg as below:

Files to be modified:

<EM_HOME>/config/IntroscopeEPAgent.properties

In "Stateful Plugins" section:

introscope.epagent.plugins.stateful.names=WEBSPHERE

introscope.epagent.stateful.WEBSPHERE.command=perl 'C:/Program Files/CA Wily/Introscope8.1.0.0/epaplugins/websphere/websphereLogReader.pl'

<EM_HOME>/epaplugins/websphere/websphereLogReader.cfg

LOGFILEDIR = 'C:/IBM/WebSphere61/AppServer/profiles/AppSrv02/logs/server1'

LOGFILE = 'SystemOut.log'

SLEEPTIME =300

{

match => 'WSVR0605W',

metricType => 'IntAverage',

metricName => 'Thread Errors',

metricValue => '$CURRENTCOUNT'

},

The paths above are for my environment on windows. Your's may be different.

b) If you are not using weblogc or websphere and you need a generic log reader you might want to use logreader_package.zip that you can customize it as you need, however code modification might be required. I have attached to the documentation library.

NOTE: Unless you know the generated code error/message in the appserver log, you cannot use EPAgent is thisfashion, otherwise you might want to use stall counts, which detects stalls of a given method.

OPTION 2: Use Stall Metrics:

The StalledMethodTracer is used to identify any "hung threads", which are threads that get stuck in a method call

StalledMethodTracer traces the number of method invocations that have exceeded a user-defined execution threshold value and have not finished. This number serves as an approximation of the number of hung threads. The result is reported in the Metrics tree using the fully qualified Metric name specified in the directives file. If desired, users can set up Alerts on this Metric.

-Stall metrics are turned on by default. If you navigate through the Investigator tree of a particular agent you will see a stall count. You might want to create a metric group based on the stall count metrics. If a stall is discovered send an alert. Stall are generally associated with a hung thread

-Stall events, on the other hand, are enabled with ErrorDetector. You will have to install and configure error detector of you are using v8.x

-Stalls are stored as ordinary errors, and will be visible in the Errors TypeView, or in the historical query viewer by querying for "type:errorsnapshot"

For more information on the StalledMethodTracer refer to the JavaAgent.pdf

OPTION 3: Use PMI data - This will give you the number of hung threads, I suggest you to discuss this with your appserver admin person.

OPTION 4: Turn on ThreadTracing - Uncomment "ThreadTracing" property in your toggles-full and toggles-typical pbd files. This will monitor each thread.