Transaction tracer issues - not catching all transactions above a particular threshold

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• CA Application Performance Management (APM / Wily / Introscope)
• INTROSCOPE
• DX Application Performance Management

Issue/Introduction

When running a transaction trace session against a 20.4 agent and we find that not as many traces are captured in the session as would be expected based on the trace threshold criteria defined.  For example the following TT session shows 4 traces captured between 08:52:56 and 08:53:49

However the investigator for the same agent shows 11 traces in the same period

Environment

Release : 20.4

Component : APM Agents

Cause

If a Sampled trace is collected for an agent whilst a transaction trace is running for that agent then that trace will not appear in the Transaction Trace window.  For example the trace highlighted above does not appear the the TT session window because it is Sampled

This is not specific to 20.4 agents – if a 10.7 agent had a Sampled trace during a TT session the behaviour would be the same

Resolution

In 20.4 the agents trace sampling frequency defaults are much higher than in previous release. In 10.7 for example the default sampling rate is defined in the IntroscopeEnterpriseManager.properties configuration by properties

 

#introscope.agent.transactiontracer.sampling.perinterval.count=1

#introscope.agent.transactiontracer.sampling.interval.seconds=120

 

In 20.4 the trace sampling is defined in the agent configuration and the default sampling rate is far higher and is defined in the

# Bundle: java-agent-base:20.4.0.17

 

Section of the IntroscopeAgent.profile

 

# Uncomment to disable Transaction Tracer Sampling.
#
# Changing this property does not require restarting the Monitored Process
# introscope.agent.transactiontracer.sampling.enabled=false

and

# Sampling strategy to use. Supported values: random (default) - pick randomly
# specified number of transactions during given time interval. firstn - sample
# first specified number of transactions during given interval.
#
# Changing this property does not require restarting the Monitored Process
#introscope.agent.transactiontracer.sampling.enabled.set=random

# Number of samples to take during given time interval for random sampling
# strategy
#
# Changing this property does not require restarting the Monitored Process
introscope.agent.transactiontracer.sampling.random.perinterval.count=12

# Duration of time interval for random sampling strategy.
#
# Changing this property does not require restarting the Monitored Process
introscope.agent.transactiontracer.sampling.random.interval.seconds=60

 

In order to prevent trace sampling during a TT session to avoid losing expected traces uncomment the following property

 

introscope.agent.transactiontracer.sampling.enabled=false

 

To set the sampling rate to the same value as 10.7 set the following values

 

introscope.agent.transactiontracer.sampling.random.perinterval.count=1 introscope.agent.transactiontracer.sampling.random.interval.seconds=120