Symptoms:
No result in dashboards when you select for example past 24 hours
AXA 21.3.1
You can use the below steps to confirm there is LAG affecting the AXA data processing:
1) Find out kafka pod names:
kubectl get pods -n<namespace>| grep kafka
2) Check for possible LAGs:
kubectl exec -ti -n<namespace> <jarvis-kafka-pod> -- /opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --describe --group axa.transformer
kubectl exec -ti -n<namespace> <jarvis-kafka-pod> -- /opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --describe --group axa-aggregator_network_perf
kubectl exec -ti -n<namespace> <jarvis-kafka-pod> -- /opt/ca/kafka/bin/kafka-consumer-groups.sh --bootstrap-server jarvis-kafka:9092,jarvis-kafka-2:9092,jarvis-kafka-3:9092 --describe --group axa-aggregator_device_perf
kubectl scale --replicas=5 deployment axaservices-transformer -n<namespace>
kubectl scale --replicas=5 deployment axaservices-indexer -n<namespace>
kubectl scale --replicas=5 deployment axaservices-kibana-indexer -n<namespace>
kubectl scale --replicas=5 deployment axaservices-axa-ng-aggregator -n<namespace>
Sample:
kafka-topics.sh --alter --zookeeper jarvis-zookeeper:2181 --topic maaBAAggregator --partitions 8
Verification:
kafka-topics.sh --describe --zookeeper jarvis-zookeeper:2181 --topic maaBAAggregator
b) The above commands are not scoped in Product documentation as it is related to open technology Kafka.
c) You must have the same number of "Partition" and "Replica Sets" for the deployments
d) Kafka Partitions are supported to increase each time but not decrease.
3) Reduce Kafka data retention from 24 hours to 1 hour
If in additional to the LAG in maaBAAggregator you notice that Kafka data is consuming disk space, then you can try reducing data retention
Verification:
cd /nfs/ca/dxi/jarvis
du -sc ./*
215188 ./apis
3279632784 ./elasticsearch
1315136 ./esutils
250332 ./indexer
0 ./jafservices
2559380580 ./kafka
3395684 ./kafka-logs
635496 ./kron
3328 ./verifier
103040 ./zookeeper
12724 ./zookeeper-logs
You can reduce kafka data retention to 1 hour for examples as below:
/opt/ca/kafka/bin/kafka-
4) Increase the # of ingress-nginx replicas:
You can simply scale the Nginx deployment replicas using kubectl
kubectl scale deployment <nginx-deployment-name> --replicas=5
Example:
kubectl scale deployment nginx-ingress-controller -n ingress-nginx --replicas=5