AIOps - How to gracefully restart all DX services and servers?

Products

• DX Operational Intelligence
• DX Application Performance Management
• CA App Experience Analytics

Issue/Introduction

We need to reboot the servers for maintenance, what are the correct steps to gracefully restart all servers

 

Environment

DX Platform 2x

Resolution

1) Stop DX Platform services 

 

cd <DX-HOME>/bin

./dx-admin.sh stop

 

2) Wait until all the pods are down. Run the following command to check the status of the pods:

kubectl get -n<your project name> pods

 

You can ignore the pods in "Completed" status

 

 

3) Perform a graceful shutdown of all the openshift or kubernetes servers

 

reboot

 

4) Once all the servers are back and running, verify the cluster, make sure all nodes are in "Ready" status:

kubectl get nodes

 

For example:

NAME                             STATUS    ROLES                  AGE       VERSION
example1.com   Ready     compute,infra,master   177d      v1.11.0+d4cacc0
example2.com   Ready     compute                       177d      v1.11.0+d4cacc0
example3.com   Ready     compute                       177d      v1.11.0+d4cacc0
...

 

5) Start DX Platform

 

cd <DX-HOME>/tools

./dx-admin.sh start

 

this process can take several minutes

 

6) Wait until all the pods are up. Run the following command to check the status of the pods:

kubectl get -n<your project name> pods

 

7) (Optional) You can safely delete all the evicted pods (if any available): 

for evicted in $(kubectl get pods | grep "Evicted" | awk '{print $1}'); do kubectl delete pod ${evicted}; done

 

 

Additional Information

DX AIOPs - Troubleshooting, Common Issues and Best Practices
https://knowledge.broadcom.com/external/article/190815