Running an SQL REST API returns "Waited 2000 ms But did not receive the response for the message"

Products

• DX Application Performance Management

Issue/Introduction

We are running an SQL REST API like:

curl -Lk -H "Authorization: Bearer $TOKEN" -H "Accept: application/json" -H "Content-Type: application/json"  http://$EM_HOST:8081/apm/appmap/apmData/query \ -d "{ \"query\" : \"select agent_host, agent_process, avg(agg_value) from metric_data where metric_path like '%CPU:Utilization%' group by agent_host, agent_process\" }" 

but get empty response, the EM logs shows the following message:

WARN --- Manager : Waited 2000 ms But did not receive the response for the message com.wily.isengard.messageprimitives.service.MessageServiceCallMessage: 

Environment

DX APM

Cause

The issue is related to some capacity issues as listed below, the Introscope EM server process is unable process data fast enough and times out causing the warning messages and empty result.

-Metric overload
-Inadequate sequential write throughput to the SmartStor disk.
-Network contention for SAN I/O
-Contention for disk resources
-Inadequate heap memory
-Inadequate disk cache memory

 

Resolution

1. Reduce the metric load.
2. Optimize the disk configuration for sequential writes.
3. Provide a dedicated disk I/O path for SmartStor.
4. Provide additional RAM memory.
5. Allocate more heap memory.
6. Configure a larger disk cache.
7. Apply filters the query:
   If you are using CLW : Add filters to reduce the dataset and increase memory
   If you are using REST API: Add date range filters 

Additional Information

See APM 10.x Troubleshooting guide and Best Practices