APM 10.7 - Downstream cross JVM traces are missing under load for transactions
search cancel

APM 10.7 - Downstream cross JVM traces are missing under load for transactions


Article ID: 199036


Updated On:


CA Application Performance Management Agent (APM / Wily / Introscope) CA Application Performance Management (APM / Wily / Introscope) INTROSCOPE DX Application Performance Management


Currently we are losing a lot of downstream transactions, majority (80-90%) don't have all expected downstream transactions, when the transaction rate is quite high. This is with 10.7 cluster and 20.4 agents. What factors affect the collection of downstream traces?


Release : 10.7.0

Component : APM Agents


Downstream traces collection can be restricted by various clamps depending on the situation and the trace type

There are general clamps which affect all trace types

On the Enterprise Manager there is an input queue clamp



If this is exceeded all other traces in that interval are dropped by the EM - you can see if this is happening by enabling DEBUG logging and looking for a message like

9/04/20 09:57:56.754 AM EDT [DEBUG] [PO:WatchedAgentPO Mailman 4] [Manager.Agent] Clamp Hit For totalMaxTTEvents Per Interval [2]. Hence exclude Traces beyond the limit.

This is a per EM clamp

The other clamp is on the agent and again relates to the number of traces per interval



In addition there are other clamps that deal with downstream traces specifically. Smart Instrumentation has Cross Process Auto Tracing which has various sustainability metrics to monitor the feature and properties to control this feature, including several clamp properties(shown with default values):

Number of transactions a downstream agent caches for automatic cross-process trace collection.

Clamps the number of automatic traces collected by Introscope per 1 minute interval. 

Max GUID cache downstream (currently hardcoded to 50)

Clamps the number of traces sent from downstream agent. The value of this clamp limit is calculated as: introscope.agent.deep.automatic.trace.clamp * introcosope.agent.deep.automatic.trace.crossprocess.downstream.clamp.multiplier