ALERT: Some images may not load properly within the Knowledge Base Article. If you see a broken image, please right-click and select 'Open image in a new tab'. We apologize for this inconvenience.

DX APM - No Data in Experience View or Map due to queue length limit was reached

book

Article ID: 232839

calendar_today

Updated On:

Products

DX Application Performance Management

Issue/Introduction

No Data in Experience View or Map as below, how to fix it?

Environment

DX Application Performance Management 20.x and onward versions

Resolution

Resolution and Troubleshooting steps:

1) Check for possible errors in Cluster Manager

Login as masteradmin
Go to Metric Browser
Check below metrics

NOTE: there can be many services instances, so you would need to review each instance

SuperDomain|apmservices|tracestore|001|Beeans|CorrelationManager|Correlation|Backends|http_//tas/tas/graph/storeAsync:Errors Per Internal

SuperDomain|apmservices|tracestore|001|Beeans|AsyncQueueManager|processJob:Average Response Time(ms)

2) Check for exceptions in the Traces log

kubectl get pods -n<namespace> | grep trace
kubectl exec -ti <trace-pod> bash -n<namespace>
cd logs
tail -f tracestore.log

...
ERROR Unable to store graph

ServiceException Graph was discarded due to queue length limit was reached


3) Check for errors in the TAS logs

NOTE: there can be many services instances, so you would need to review each instance

kubectl get pods -n<namespace> | grep tas
kubectl exec -ti <tas-pod> bash -n<namespace>
cd logs
tail -f tas.log

if messages are not useful, unzip the previous zipped log:
gunzip tas.log.<timestamp>.0.gz
more tas.log.<timestamp>.0

result: the same ERROR is reported

4) Increase ASYNC_QUEUE_SIZE

kubectl -n<namespace> edit configmap apmservices-tas

Add the below 2 lines:

data:
  APM_TAS_ASYNC_QUEUE_SIZE: '1000000000'
  
Save the file, here is an example:

 

 

then verify the new key has been added 

 

5) Restart TAS pod

kubectl -n<namespace> delete po <tapmservices-tas-pod>

kubectl -n<namespace> logs <tapmservices-tas-pod>

6) Check if data is getting processed in Cluster Manager

In APM

Additional Information

https://knowledge.broadcom.com/external/article/190815/aiops-troubleshooting-common-issues-and.html

Attachments