Restarting DX O2 Jarvis services?

Products

• DX Operational Observability

Issue/Introduction

Some DX O2 Jarvis services are not performing as expected , some are in red as per DX O2 Jarvis Troubleshooting 

What is the process to restart Jarvis service?

Environment

DX O2

Resolution

Restart the jarvis services as below:

Scale down:
- all jarvis-kafka
- all zookeeper
- jarvis-verifier                
- jarvis-lean-jarvis-indexer 
- jarvis-indexer
- jarvis-kron
- jarvis-esutils

Scale up:
- all zookeeper
- all jarvis-kafka
- jarvis-verifier                
- jarvis-lean-jarvis-indexer 
- jarvis-indexer
- jarvis-kron
- jarvis-esutils

 

Below an example of step by step instructions in case you have 3 Elastic Nodes:

1) Scale down the following deployments:

kubectl scale --replicas=0 deployment jarvis-kafka -n<namespace>
kubectl scale --replicas=0 deployment jarvis-kafka-2 -n<namespace>
kubectl scale --replicas=0 deployment jarvis-kafka-3 -n<namespace>
kubectl scale --replicas=0 deployment jarvis-zookeeper -n<namespace>
kubectl scale --replicas=0 deployment jarvis-zookeeper-2 -n<namespace>
kubectl scale --replicas=0 deployment jarvis-zookeeper-3 -n<namespace>
kubectl scale --replicas=0 deployment jarvis-verifier -n<namespace>
kubectl scale --replicas=0 deployment jarvis-lean-jarvis-indexer  -n<namespace>
kubectl scale --replicas=0 deployment jarvis-indexer -n<namespace>
kubectl scale --replicas=0 deployment jarvis-kron -n<namespace>
kubectl scale --replicas=0 deployment jarvis-esutils -n<namespace>

2) Verify that all pods are down:

kubectl get pods -n<namespace> | egrep "jarvis-kafka|jarvis-zookeeper|jarvis-verifier|jarvis-lean|jarvis-indexer|jarvis-kron|jarvis-esutils"

3) Start pods in below order

a) scale up zookeeper pods, review the pod logs using " kubectl logs <pod-name> -n<namespace> " and verify that no errors are reported

kubectl scale --replicas=1 deployment jarvis-zookeeper -n<namespace>
kubectl scale --replicas=1 deployment jarvis-zookeeper-2 -n<namespace>
kubectl scale --replicas=1 deployment jarvis-zookeeper-3 -n<namespace>

b) Scale up kafka pods, review the pod logs 

kubectl scale --replicas=1 deployment jarvis-kafka -n<namespace>
kubectl scale --replicas=1 deployment jarvis-kafka-2 -n<namespace>
kubectl scale --replicas=1 deployment jarvis-kafka-3 -n<namespace>

c) Start the below pods, Review the pod logs

kubectl scale --replicas=1 deployment jarvis-verifier -n<namespace>
kubectl scale --replicas=1 deployment jarvis-lean-jarvis-indexer -n<namespace>
kubectl scale --replicas=1 deployment jarvis-indexer -n<namespace>
kubectl scale --replicas=1 deployment jarvis-kron -n<namespace>
kubectl scale --replicas=1 deployment jarvis-esutils -n<namespace>

4) Verify that all pods are up and running:

kubectl get pods -n<namespace> | egrep "jarvis-kafka|jarvis-zookeeper|jarvis-verifier|jarvis-lean|jarvis-indexer|jarvis-kron|jarvis-esutils"

5) Verification

Check that all Jarvis Services are in green : http(s)://doi-nginx.<endpoint>/health

For more information see DX O2 Jarvis Troubleshooting