AIOps - How to clear pods in Terminating status

Products

• DX Operational Intelligence
• DX Application Performance Management

Issue/Introduction

Sometimes pods do not stop as expected. In below example doi-situations-0 pod remains in "Terminating" status forever

How can I delete or terminate a pod in Terminating status?

Environment

DX Platform any version

Cause

Possible Reasons:

 

1) The pod has a finalizer associated with it that is not completing

2) The pod is not responding to termination signals sent by Kubernetes API Server
3) The node on which a stateful-set pod was running is unreachable
4) From: https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/ 

"A Pod is not deleted automatically when a node is unreachable. The Pods running on an unreachable Node enter the 'Terminating' or 'Unknown' state after a timeout. Pods may also enter these states when the user attempts graceful deletion of a Pod on an unreachable Node. The only ways in which a Pod in such a state can be removed from the apiserver are as follows:

The Node object is deleted (either by you, or by the Node Controller).
The kubelet on the unresponsive Node starts responding, kills the Pod and removes the entry from the apiserver.
Force deletion of the Pod by the user."

5)Check the "Events section" from the command: kubectl describe pod <PODNAME> --namespace <NAMESPACE>

 

Resolution

You can remove the pod using below command, a new pod will be created:

 

kubectl delete pod <PODNAME> --grace-period=0 --force --namespace <NAMESPACE>

 

 

Additional Information

https://knowledge.broadcom.com/external/article/190815/aiops-troubleshooting-common-issues-and.html