search cancel

"AXA Transformer" at 400% cpu and App sessions appears but after several hours

book

Article ID: 238179

calendar_today

Updated On:

Products

CA App Experience Analytics

Issue/Introduction

Symptoms:

- Data from applications is visible in AXA but appears after several hours

- "Cluster management" reports "AXA Transformer" CPU at 400% 

How can we solve the issue?

Cause

Not enough capacity to process all incoming AXA data

 

How to verify this condition?

1) connect to a kafka pod:

 kubectl exec -ti <jarvis-kafka-pod> sh -n<namepsace>


2) Check if there is a LAG processing the AXA data

 /opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --describe --group axa.transformer

Environment

DX AXA 20.x, 21.x

Resolution

1) Increase the # of replicas for axa services as below:

kubectl scale --replicas=4 deployment axaservices-transformer -n<namespace>
kubectl scale --replicas=4 deployment axaservices-indexer -n<namespace>
kubectl scale --replicas=4 deployment axaservices-kibana-indexer -n<namespace>
kubectl scale --replicas=4 deployment axaservices-axa-user-processor -n<namespace>
kubectl scale --replicas=2 deployment axaservices-axa-ng-aggregator -n<namespace>


2) Connect to a kafka pod and verify if the LAG issue is resolved

 

NOTE: You can increase the # of replicas of the above deployments to a maximum of 5, if the problem persist after this, contact Broadcom Support

Additional Information

https://knowledge.broadcom.com/external/article/190815/aiops-troubleshooting-common-issues-and.html

Attachments