Elastic search calls not showing up.

Products

• CA Application Performance Management Agent (APM / Wily / Introscope)
• CA Application Performance Management (APM / Wily / Introscope)
• INTROSCOPE
• DX Application Performance Management

Issue/Introduction

We have a Tomcat instance instrumented with APM agent Release 20.1.0.44 (Build 990044). The Tomcat server makes API calls to an elastic search backend. But even with the application hits, we don't see elastic search backend calls being shown in CA APM tool. 

 

Environment

Release : 14.6

Component : APM Agents

Cause

Need for custom tracers.

Resolution

 The autoprobe log didn't show any instrumentation from elasticsearch-transport.pbd being applied to the monitored app. The elasticsearch-transport.pbd specifies instrumentations on the following classes/methods to report metrics under metric path "NotReq|{ElasticSearch}|{index}|{operation}", "{ElasticSearch}" and "{ElasticSearch}|{index}|{operation}":



Class: org.elasticsearch.transport.TransportService

Method:

sendRequest(Lorg/elasticsearch/cluster/node/DiscoveryNode;Ljava/lang/String;Lorg/elasticsearch/transport/TransportRequest;Lorg/elasticsearch/transport/TransportRequestOptions;Lorg/elasticsearch/transport/TransportResponseHandler;)V



Class: org.elasticsearch.action.ActionFuture

Methods:

actionGet()Ljava/lang/Object

actionGet(JLjava/util/concurrent/TimeUnit;)Ljava/lang/Object



The customer appeared not to be using or extending these classes/methods. If so, they need to customize the pbd to cover the specific elastic search implementations used in their apps

Additional Information

Tracers that solved the issue and customer can see custom Elasticsearch methods showing metrics

TurnOn: ESRestHLClientBackendTracing

IdentifyClassAs: org.elasticsearch.client.RestHighLevelClient ESRestHLClientBackendTracing

TraceOneMethodWithParametersIfFlagged: ESRestHLClientBackendTracing search(Lorg/elasticsearch/action/search/searchRequest;Lorg/elasticsearch/client/RequestOptions;)Lorg/elasticsearch/action/search/SearchResponse; BlamePointTracer "Custom|RestHLClient|search"

 

To detect similar methods as shown by the other APM, we could consider the following changes in ElasticSearch.pbd:

 

SetFlag: ESRestClientTracing
TurnOn: ESRestClientTracing

 

IdentifyClassAs: org.elasticsearch.client.RestClient ESRestClientTracing

TraceOneMethodIfFlagged: ESRestClientTracing performRequest BlamePointTracer "Custom|ElasticSearch|{classname}|{method}"

 

SetFlag: ESRestHLClientBackendTracing
TurnOn: ESRestHLClientBackendTracing

 

IdentifyClassAs: org.elasticsearch.client.RestHighLevelClient ESRestHLClientBackendTracing

TraceOneMethodIfFlagged: ESRestHLClientBackendTracing internalPerformRequest BlamePointTracer "Custom|ElasticSearch|{classname}|{method}"

TraceOneMethodIfFlagged: ESRestHLClientBackendTracing performRequest BlamePointTracer "Custom|ElasticSearch|{classname}|{method}"

TraceOneMethodIfFlagged: ESRestHLClientBackendTracing performRequestAndParseEntity BlamePointTracer "Custom|ElasticSearch|{classname}|{method}"

TraceOneMethodIfFlagged: ESRestHLClientBackendTracing search BlamePointTracer "Custom|ElasticSearch|{classname}|{method}"