Safe downtime for workerX without effecting the AIOPS

Products

• DX Operational Intelligence

Issue/Introduction

How to safely shutdown workerX for operation  say  for 5 mins .

Environment

Release : 21.3

Resolution

Steps are as below

1:  Run Drain commmand and ignore that two pods

2:  Check if AIOPS all pods are up and running

if above two are ok 

Then shutdown the node

3: post the node is up run the uncordon command  so that node is again in schedulable  state.

 

Below are the  Example of the command

**************************************************

>oc adm drain ca-workerX --force --ignore-daemonsets=true --delete-emptydir-data=true

>oc get pods -o wide | grep workerX ----> output as below

apmservices-hostmonitor-YYYYY                        1/1     Running     0          4d3h   10.131.5.63    ca-workerX              <none>           <none>

haveged-YYYY                                       1/1     Running     0          4d3h   10.131.5.62    ca-workerX              <none>           <none>

Above  two pods will not move. you can ignore that 

Post that once you run the uncordon command ,the pods will not retrun back immediately. It is like node is schedulable so cluster can assige back the pod as needed

Post running the  command if all pods are running state , that means it worked fine