pods on ContainerCreating status

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

   Some pods are on ContainerCreating state:

root@k8cluster-xxxxx-xxxxx [ ~ ]# kubectl get po -n kube-system -owide
NAME                                                    READY   STATUS              RESTARTS      AGE     IP            NODE                                    NOMINATED NODE   READINESS GATES
antrea-agent-xxxxx                                      2/2     Running             0             164d    xxx.xx.xxx.27    k8clusterprd-worker-xxxxx-xxxxx-xxxxx   <none>           <none>
antrea-agent-xxxxx                                      2/2     Running             0             264d    xxx.xx.xxx.24    k8clusterprd-xxxxx-xxxxx                <none>           <none>
antrea-agent-xxxxx                                      0/2     Init:0/2            0             45s     xxx.xx.xxx.28    k8clusterprd-worker-xxxxx-xxxxx-xxxxx   <none>           <none>
antrea-controller-xxxxxxxxxx-xxxxx                      1/1     Running             0             5h36m   xxx.xxx.xxx.27   k8clusterprd-worker-xxxxx-xxxxx-xxxxx   <none>           <none>
coredns-xxxxxxxxxx-xxxxx                                1/1     Running             0             264d    xxx.xxx.xxx.7    k8clusterprd-xxxxx-xxxxx                <none>           <none>
coredns-xxxxxxxxxx-xxxxx                                1/1     Running             0             264d    xxx.xxx.xxx.4    k8clusterprd-xxxxx-xxxxx                <none>           <none>
docker-registry-k8cluster-xxxxx-6r8cp                   1/1     Running             0             264d    xxx.xxx.xxx.24   k8clusterprd-xxxxx-xxxxx                <none>           <none>
docker-registry-k8cluster-worker-xxxxx-xxxxx-xxxxx      0/1     ContainerCreating   0             164d    xxx.xxx.xxx.28   k8clusterprd-worker-xxxxx-xxxxx-xxxxx   <none>           <none>
docker-registry-k8cluster-worker-xxxxx-xxxxx-xxxxx      1/1     Running             0             164d    xxx.xxx.xxx.27   k8clusterprd-worker-xxxxx-xxxxx-xxxxx   <none>           <none>
etcd-k8cluster-xxxxx-xxxxx                              1/1     Running             0             264d    xxx.xxx.xxx.24   k8clusterprd-xxxxx-xxxxx                <none>           <none>
kube-apiserver-k8cluster-xxxxx-xxxxx                    1/1     Running             0             264d    xxx.xxx.xxx.24   k8clusterprd-xxxxx-xxxxx                <none>           <none>
kube-controller-manager-k8cluster-xxxxx-xxxxx           1/1     Running             17 (6d ago)   264d    xxx.xxx.xxx.24   k8clusterprd-xxxxx-xxxxx                <none>           <none>
kube-proxy-xxxxx                                        1/1     Running             0             164d    xxx.xxx.xxx.27   k8clusterprd-worker-xxxxx-xxxxx-xxxxx   <none>           <none>
kube-proxy-xxxxx                                        1/1     Running             0             264d    xxx.xxx.xxx.24   k8clusterprd-xxxxx-xxxxx                <none>           <none>
kube-proxy-xxxxx                                        0/1     ContainerCreating   1             164d    xxx.xxx.xxx.28   k8clusterprd-worker-xxxxx-xxxxx-xxxxx   <none>           <none>
kube-scheduler-k8cluster-xxxxx-xxxxx                    1/1     Running             17 (6d ago)   264d    xxx.xxx.xxx.24   k8clusterprd-xxxxx-xxxxx                <none>           <none>
metrics-server-xxxxxxxxxx-xxxxx                         0/1     ContainerCreating   1             164d    xxx.xxx.xxx.28   k8clusterprd-worker-xxxxx-xxxxx-xxxxx   <none>           <none>
snapshot-controller-xxxxxxxxxx-xxxxx                    1/1     Running             1 (27d ago)   264d    xxx.xxx.xxx.2    k8clusterprd-xxxxx-xxxxx                <none>           <none>


 

   * All failing pods are attached to the same nod

 

   Cluster nodes looks apparently healthy:

root@k8cluster-xxxxx-xxxxx  [ ~ ]# kubectl get node
NAME                                    STATUS   ROLES           AGE    VERSION
k8cluster-xxxx-xxxxx                 Ready    control-plane   264d   v1.30.1+vmware.1-fips
k8cluster-worker-xxxxx-xxxxx-xxxxx   Ready    <none>          164d   v1.30.1+vmware.1-fips
k8cluster-worker-xxxxx-xxxxx-xxxxx   Ready    <none>          164d   v1.30.1+vmware.1-fips

 

   Pod description events:

root@k8cluster-xxxxx-xxxxx [ ~ ]# kubectl describe po -n rtf agent-xxxxxxxxxx-xxxxx
Events:
  Type     Reason                  Age                    From     Message
  ----     ------                  ----                   ----     -------
  Warning  FailedCreatePodSandBox  81s (x1398 over 5h6m)  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "localhost:5000/vmware.io/pause:3.9": failed to pull image "localhost:5000/vmware.io/pause:3.9": failed to pull and unpack image "localhost:5000/vmware.io/pause:3.9": failed to resolve reference "localhost:5000/vmware.io/pause:3.9": failed to do request: Head "http://localhost:5000/v2/vmware.io/pause/manifests/3.9": dial tcp 127.0.0.1:5000: connect: connection refused
  

   Containerd and kubelet are showing errors on that node:

root@k8cluster-xxxxx-xxxxx [ ~ ]# journalctl -xeu containerd

MM DD HH:mm:ss k8cluster-xxxxx-xxxxx containerd[879]: time="YYYY-MM-DDTHH:mm:ss.630878337Z" level=info msg="Executing port forwarding in network namespace \"host\""
MM DD HH:mm:ss k8cluster-xxxxx-xxxxx containerd[879]: EMMDD HH:MM:ss.637467     879 httpstream.go:290] error forwarding port 2379 to pod xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx, uid : failed to execute portforward in network namespace "host": EOF
MM DD HH:mm:ss k8cluster-xxxxx-xxxxx containerd[879]: time="YYYY-MM-DDTHH:mm:ss.640626414Z" level=info msg="Finish port forwarding for \"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\" port 2379"

root@k8cluster-xxxxx-xxxxx [ ~ ]# journalctl -xeu kubelet
MM DD HH:mm:ss k8cluster-xxxxx-xxxxx kubelet[1485]: EMMDD HH:mm:ss.993236    1485 upgradeaware.go:427] Error proxying data from client to backend: readfrom tcp 127.0.0.1:38800->127.0.0.1:44141: read tcp xxx.xxx.xxx.24:10250->xxx.xxx.xxx.24:55324: read: connection reset by peer
MM DD HH:mm:ss k8cluster-xxxxx-xxxxx kubelet[1485]: EMMDD HH:mm:ss.328427    1485 upgradeaware.go:427] Error proxying data from client to backend: readfrom tcp 127.0.0.1:46134->127.0.0.1:44141: read tcp xxx.xxx.xxx.24:10250->xxx.xxx.xxx.24:34366: read: connection reset by peer

 

 

.

Cause

Node service degradation.

Resolution

Restart the node by vCenter UI. Stop and start it.

After restarting the node, make sure all pods on that node are running.