DX APM - If ElasticSearch node#1 is shutdown "Trace tab" reports No Data

Products

• DX Application Performance Management

Issue/Introduction

While testing redundancy, we shutdown Elastic Node 1 and now Traces tab in APM reports No Data

 

Environment

DX APPLICATION PERFORMANCE MANAGEMENT (APM) 20.x

Cause

This problem is related to defect DE517399:

Product deployments are configured to use ElasticSearch Node#1 instead to ElasticSearch LoadBalancer

Resolution

Below are the steps how to reconfigure DX APM components to point to ElasticSearch Load Balancer:

1) Update apm-em deployments

a) List all apm-em deployments:

 

kubectl get deployments -n<namespace> | grep apm-em

 

b) Edit all apm-em deployments :

kubectl edit deployment apm-em -n<namespace>

c) Replace 'jarvis-elasticsearch'  to 'jarvis-elasticsearch-lb' for below key :

• ca.apm.ttstore.elastic.url

NOTE: After saving the changes new ones will be created

 

 

2) Update apmservices-tracestore config map

 

a) Open apmservices-tracestore config maps using:

 

kubectl edit cm apmservices-tracestore -n<namespace>

 

b) Replace 'jarvis-elasticsearch'  to 'jarvis-elasticsearch-lb' for below key:

• APM_TRACESTORE_ELASTICURL 

 

Additional Information

https://knowledge.broadcom.com/external/article/190815/dx-aiops-troubleshooting-common-issues-a.html