DX APM - No Data in Experience View or Map due to queue length limit was reached
Article ID: 232839
Updated On:
Products
Issue/Introduction
No Data in Experience View or Map as below, how to fix it?
Environment
DX Application Performance Management 20.x and onward versions
Resolution
Table of Contents
Resolution and Troubleshooting steps:
1) Check for possible errors in Cluster Manager
Login as masteradmin
Go to Metric Browser
Check below metrics
NOTE: there can be many services instances, so you would need to review each instance
SuperDomain|apmservices|tracestore|001|Beeans|CorrelationManager|Correlation|Backends|http_//tas/tas/graph/storeAsync:Errors Per Internal
SuperDomain|apmservices|tracestore|001|Beeans|AsyncQueueManager|processJob:Average Response Time(ms)
2) Check for exceptions in the Traces log
kubectl get pods -n<namespace> | grep trace
kubectl exec -ti <trace-pod> bash -n<namespace>
cd logs
tail -f tracestore.log
...
ERROR Unable to store graph
ServiceException Graph was discarded due to queue length limit was reached
3) Check for errors in the TAS logs
NOTE: there can be many services instances, so you would need to review each instance
kubectl get pods -n<namespace> | grep tas
kubectl exec -ti <tas-pod> bash -n<namespace>
cd logs
tail -f tas.log
if messages are not useful, unzip the previous zipped log:
gunzip tas.log.<timestamp>.0.gz
more tas.log.<timestamp>.0
result: the same ERROR is reported
4) Increase ASYNC_QUEUE_SIZE
kubectl -n<namespace> edit configmap apmservices-tas
Add the below 2 lines:
data:
APM_TAS_ASYNC_QUEUE_SIZE: '1000000000'
Save the file, here is an example:
then verify the new key has been added
5) Restart TAS pod
kubectl -n<namespace> delete po <tapmservices-tas-pod>
kubectl -n<namespace> logs <tapmservices-tas-pod>
6) Check if data is getting processed in Cluster Manager
In APM
Additional Information
Feedback
