DX AXA - App sessions are not being reported, How to increase the AXA Transformer limit of 5 instances?

Products

• DX Operational Intelligence
• CA App Experience Analytics
• DX Application Performance Management

Issue/Introduction

Observations:

 

1) AXA app sessions are not being reported:

 

2) AXA Transformer instances have been increased to the max of 5 instances, from Cluster Management >  Services

 

3)  When troubleshooting kafka, we can see lags in some axa transformer partitions

 

Steps taken to verify this condition:

kubectl get pods -n<dxi-namespace> | grep kafka
kubectl exec -ti <kafka-pod> sh -n<dxi-namespace>

/opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --describe --group axa.transformer

..
axa.transformer maaBAAggregator         4          64818420        64818481        61              JsonMessageProcessor_maaBAAggregator_axaservices-transformer-ccc9b76f7-z52sx-dc2e9e86-52ab-445c-8c5a-51b782246628            /#.#.#.#  JsonMessageProcessor_maaBAAggregator_axaservices-transformer-ccc9b76f7-z52sx

axa.transformer maaBAAggregator         2          1345066957      1345864329      797372          JsonMessageProcessor_maaBAAggregator_axaservices-transformer-ccc9b76f7-g6rw6-b87c4b22-3ee9-4b3c-8cb8-eefd97d3d830            /#.#.#.#  JsonMessageProcessor_maaBAAggregator_axaservices-transformer-ccc9b76f7-g6rw6

axa.transformer maaBAAggregator         0          63974072        64817107        843035          JsonMessageProcessor_maaBAAggregator_axaservices-transformer-ccc9b76f7-67jjd-79375602-ddb9-49c2-bd73-5dbe3570f750            /#.#.#.#  JsonMessageProcessor_maaBAAggregator_axaservices-transformer-ccc9b76f7-67jjd

axa.transformer maaBAAggregator         1          65068075        65068161        86              JsonMessageProcessor_maaBAAggregator_axaservices-transformer-ccc9b76f7-cnxqr-97e5c7ae-71e6-430f-8f93-edd0024f17bc            /#.#.#.#  JsonMessageProcessor_maaBAAggregator_axaservices-transformer-ccc9b76f7-cnxqr

axa.transformer maaBAAggregator         3          64817336        64817416        80              JsonMessageProcessor_maaBAAggregator_axaservices-transformer-ccc9b76f7-tthnb-88aa7fdc-b17e-442a-a268-6c48cc400d02            /#.#.#.#  JsonMessageProcessor_maaBAAggregator_axaservices-transformer-ccc9b76f7-tthnb
...

 

see also "AXA Transformer" at 400% cpu and App sessions appears but after several hours

 

 

How to increase the # of AXA Transformer instances more than 5? 

Environment

DX Platform 2x.x

Resolution

You can change the default maximum # of instances as below:

STEP # 1: Increasing the # of partitions
STEP # 2: Increasing the # of AXA Transformer instances
STEP # 3: Scale up other components (User Processor, Indexer, Kibana Indexer and NG-Aggregator) to keep overall health/controlled lag across all queues .
STEP # 4: (Optional) Apply 21.3.1 HF3 that solve defect " Parallelism in DXC and AXA Decryptor' : Upgrade 21.3.1.HF1 to 21.3.1.HF3 

 

STEP # 1: Increase the # of partitions

kafka-topics.sh --alter --zookeeper jarvis-zookeeper:2181 --topic <TOPIC NAME> --partitions <PartitionsCount>

Example:
To increase the # partition from max of 5 to 8:
kafka-topics.sh --alter --zookeeper jarvis-zookeeper:2181 --topic maaBAAggregator --partitions 8

IMPORTANT:

1) These commands are not scoped in Product documentation as it is related to open technology Kafka.
2) Recommendation is to have the same number of partition and Replica Sets for the deployment
3) Kafka Partitions are supported to increase each time but not decrease.

Verification:

To verify the change you can describe the topic as below:
kafka-topics.sh --describe --zookeeper jarvis-zookeeper:2181 --topic maaBAAggregator

 

STEP # 2: Increase the # of AXA Transformer instances:

kubectl scale --replicas=<REPLICAS #> deployment axaservices-transformer -n <namespace>

For example:
kubectl scale --replicas=8 deployment axaservices-transformer -ndxi

 

STEP # 3: Scale up other components (User Processor, Indexer, Kibana Indexer and NG-Aggregator) to keep overall health/controlled lag across all queues

kubectl scale --replicas=8 deployment axaservices-transformer -n<namespace>
kubectl scale --replicas=8 deployment axaservices-indexer -n<namespace>
kubectl scale --replicas=8 deployment axaservices-kibana-indexer -n<namespace>
kubectl scale --replicas=8 deployment axaservices-axa-user-processor -n<namespace>
kubectl scale --replicas=4 deployment axaservices-axa-ng-aggregator -n<namespace>

See also: "AXA Transformer" at 400% cpu and App sessions appears but after several hours

Additional Information

AIOPs - Troubleshooting, Common Issues and Best Practices