Traces processing error

Products

• DX Application Performance Management

Issue/Introduction

Issue Troubleshooting: Trace processing/missing 

How can we check what is wrong in the environment?

Symptoms:

• Metric view: No traces for the application agents.
• Map view: No data on application layer, except WEBSERVICEs components. The data expires in 24 hours when trace processing stops
• Experience view has no data. Also after 24 hours after trace processing is off

Environment

DX O2 SaaS
DX O2 25.1

Resolution

These components are involved in trace processing:

apmservices.tracesore:

The component is responsible for processing trace files and creating map components.
● Component is running
● Log files
● Metrics in Cluster Manager: under masteradmin login session
    ○ apmservices|tracestore|001|Resources|Memory|*,apmservices|tracestore|001|Resources|CPU|*: whether it is hitting memory or CPU limits
    ○ apmservices|tracestore|001|Beans|TTProcessor:Fast TransactionsProcessed, apmservices|tracestore|001|Beans|TTProcessor:SlowTransactions Processed: whether it is receiving/processing Traces 
When empty: apmservices.cloudgw could be the issue
   ○ apmservices|tracestore|001|Beans|ModelSynchronization:SentVertices Per Interval: whether it creating any map components
When empty: apmservices.tas could be the issue
apmservices|tracestore|001|Elastic Trace Store:Traces Stored PerInterval: whether it is storing processed traces
When empty: elasticsearch could be the issue

apmservices.cloudgw:

The component is responsible for obtaining data from agents (traces and metrics) and sending
traces to apmservice.tracestore
● Component is running
● Log files
● Metrics in Cluster Manager: under masteradmin login session
  ○ apmservices|cloudgw|001|Gateway|Traces|Tenants|:Traces Count Per Interval: whether it receives traces from agents for the tenant
When empty: agents could be the issue