APM Agent - Exclude http.clients from _nodes/stats in elasticsearch schema

Products

• DX APM SaaS

Issue/Introduction

The AIOPs APM agents are failing to pull metrics from elasticsearch clusters. 

Error Details:

[IntroscopeAgent.RESTMon] Exception in: Elastica_INVESTIGATE_C3_Logs~~elasticsearch. Will attempt to re-establish database connection.
java.lang.NullPointerException
at com.ca.ce.restmon.util.ParserUtil.addTokenMaptoBindings(ParserUtil.java:98)
at com.ca.ce.restmon.util.ParserUtil.parseForDataContext(ParserUtil.java:79)
at com.ca.ce.restmon.core.ResponseUrlTokenHandler.collectUrlTokens(ResponseUrlTokenHandler.java:56)
at com.ca.ce.restmon.core.ResponseUrlTokenHandler.handleUrlTokens(ResponseUrlTokenHandler.java:38)
at com.ca.ce.restmon.core.DataCollector.insertIntoDataTable(DataCollector.java:685)
at com.ca.ce.restmon.core.DataCollector.executeUrlTasks(DataCollector.java:503)
at com.ca.ce.restmon.core.DataCollector.collectEmptyUrlsData(DataCollector.java:434)
at com.ca.ce.restmon.core.DataCollector.collectAllDataAndInsertIntoDB(DataCollector.java:83)
at com.ca.ce.restmon.core.ProfileHandler.processProfile(ProfileHandler.java:205)
at com.ca.apm.agent.extension.restmon.core.MetricCollector.run(MetricCollector.java:97)
at com.ca.apm.agent.extension.restmon.core.MetricCollector.call(MetricCollector.java:79)
at com.ca.apm.agent.extension.restmon.core.MetricCollector.call(MetricCollector.java:1)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

 

Issue: APM agent failing to parse huge http.clients information from ES 7.17 clusters

Resolution

We are able to resolve this issue by recreating image of the agent with below fix.

/usr/local/apmia/extensions/Elasticsearch/config/elasticsearch_schema.json{"src": "","xml_ns": "","var": "","id": "nodestats","url": "/_nodes/stats?filter_path=-nodes.*.http.clients,-nodes.*.discovery","tokens": {"clusterName": "$['cluster_name']"}}

ER request filed as above is a workaround