Pod stays in ContainerCreating state and does not progress

Products

• VMware Telco Cloud Automation

Issue/Introduction

You might run into an issue where a pod will go in ContainerCreating state and will stay in this state. Describing the pod you will see this message:

Warning  FailedMount  2m52s (x41 over 93m)  kubelet  (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[ABC], unattached volumes=[ABC XYZ]: timed out waiting for the condition

Environment

2.x

Resolution

When checking all the PVs and volumeattchments, everything seems to be there, it is just that the pod is not able to attach the volume for some reason and is timing out.

Check the node cpu consumption by going into the node and run top. In this case the cpu utilization was quite high:

top - 17:26:28 up 231 days,  9:22,  2 users,  load average: 3658.64, 3657.45, 3654.89

Drian the node, so the pod can be schedule on some other node not heavily utilized in terms of CPU