App sessions are not being reported

Products

• DX Operational Intelligence
• CA App Experience Analytics
• DX Application Performance Management
• DX Operational Observability

Issue/Introduction

Symptoms:

 

1) AXA app sessions are not being reported:

 

2) AXA Transformer instances have been increased to the max of 5 instances, from Cluster Management >  Services

 

3)  When troubleshooting kafka, we can see lags in some axa transformer partitions

 

Steps taken to verify this condition:

kubectl get pods -n<dxi-namespace> | grep kafka
kubectl exec -ti <kafka-pod> sh -n<dxi-namespace>

/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --describe --group axa.transformer

..
axa.transformer maaBAAggregator         4          64818420        64818481        61              JsonMessageProcessor_maaBAAggregator_axaservices-transformer
axa.transformer maaBAAggregator         2          1345066957    1345864329    797372   JsonMessageProcessor_maaBAAggregator_axaservices-transformer
axa.transformer maaBAAggregator         0          63974072        64817107        843035   JsonMessageProcessor_maaBAAggregator_axaservices-transformer
axa.transformer maaBAAggregator         1          65068075        65068161        86              JsonMessageProcessor_maaBAAggregator_axaservices-transformer
axa.transformer maaBAAggregator         3          64817336        64817416        80              JsonMessageProcessor_maaBAAggregator_axaservices-transformer

 

 

How to increase the # of AXA Transformer instances more than 5? 

Environment

DX O2 2*

Resolution

Table of Contents

• STEP # 1: Increase the # of partitions
• STEP # 2: Increase the # of AXA Transformer instances:
• STEP # 3: Scale up other components (User Processor, Indexer, Kibana Indexer and NG-Aggregator) to keep overall health/controlled lag across all queues

STEP # 1: Increase the # of partitions

kafka-topics.sh --alter --zookeeper jarvis-zookeeper:2181 --topic <TOPIC NAME> --partitions <PartitionsCount>

Example:
To increase the # partition from max of 5 to 8:
kafka-topics.sh --alter --zookeeper jarvis-zookeeper:2181 --topic maaBAAggregator --partitions 8

IMPORTANT:

1) These commands are not scoped in Product documentation as it is related to open technology Kafka.
2) Recommendation is to have the same number of partition and Replica Sets for the deployment
3) Kafka Partitions are supported to increase each time but not decrease.

Verification:

To verify the change you can describe the topic as below:
kafka-topics.sh --describe --zookeeper jarvis-zookeeper:2181 --topic maaBAAggregator

 

STEP # 2: Increase the # of AXA Transformer instances:

kubectl scale --replicas=<REPLICAS #> deployment axaservices-transformer -n <namespace>

For example:
kubectl scale --replicas=8 deployment axaservices-transformer -ndxi

 

STEP # 3: Scale up other components (User Processor, Indexer, Kibana Indexer and NG-Aggregator) to keep overall health/controlled lag across all queues

kubectl scale --replicas=8 deployment axaservices-transformer -n<namespace>
kubectl scale --replicas=8 deployment axaservices-indexer -n<namespace>
kubectl scale --replicas=8 deployment axaservices-kibana-indexer -n<namespace>
kubectl scale --replicas=8 deployment axaservices-axa-user-processor -n<namespace>
kubectl scale --replicas=4 deployment axaservices-axa-ng-aggregator -n<namespace>

See also: "AXA Transformer" at 400% cpu and App sessions appears but after several hours