APM 10.7 - Downstream cross JVM traces are missing under load for transactions

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• CA Application Performance Management (APM / Wily / Introscope)
• INTROSCOPE
• DX Application Performance Management

Issue/Introduction

Currently we are losing a lot of downstream transactions, majority (80-90%) don't have all expected downstream transactions, when the transaction rate is quite high. This is with 10.7 cluster and 20.4 agents. What factors affect the collection of downstream traces?

Environment

Release : 10.7.0

Component : APM Agents

Resolution

Downstream traces collection can be restricted by various clamps depending on the situation and the trace type

There are general clamps which affect all trace types

On the Enterprise Manager there is an input queue clamp

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/application-performance-management/10-7/administrating/properties-files-reference/apm-events-thresholds-config-xml.html

introscope.enterprisemanager.events.limit=1250

If this is exceeded all other traces in that interval are dropped by the EM - you can see if this is happening by enabling DEBUG logging and looking for a message like

9/04/20 09:57:56.754 AM EDT [DEBUG] [PO:WatchedAgentPO Mailman 4] [Manager.Agent] Clamp Hit For totalMaxTTEvents Per Interval [2]. Hence exclude Traces beyond the limit.

This is a per EM clamp

The other clamp is on the agent and again relates to the number of traces per interval

https://techdocs.broadcom.com/us/en/ca-enterprise-software/it-operations-management/application-performance-management/20-2/implementing-agents/java-agent/java-agent-properties.html#concept.dita_e3b0b5e4c888550feb742b0f947b9fa2e2fd716d_TransactionTracingProperties

introscope.agent.ttClamp=50

In addition there are other clamps that deal with downstream traces specifically. Smart Instrumentation has Cross Process Auto Tracing which has various sustainability metrics to monitor the feature and properties to control this feature, including several clamp properties(shown with default values):

introscope.agent.deep.automatic.trace.crossprocess.transactions.cache.max=100
Number of transactions a downstream agent caches for automatic cross-process trace collection.

introscope.agent.deep.automatic.trace.clamp=10
Clamps the number of automatic traces collected by Introscope per 1 minute interval. 

introcosope.agent.deep.automatic.trace.crossprocess.downstream.clamp=50
Max GUID cache downstream (currently hardcoded to 50)

introcosope.agent.deep.automatic.trace.crossprocess.downstream.clamp.multiplier=10
Clamps the number of traces sent from downstream agent. The value of this clamp limit is calculated as: introscope.agent.deep.automatic.trace.clamp * introcosope.agent.deep.automatic.trace.crossprocess.downstream.clamp.multiplier