Inaccurate container status reported By Containerd in TKGm workload cluster

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform
• VMware Telco Cloud Platform - 5G Edition
• VMware Telco Cloud Platform Advanced
• VMware Telco Cloud Platform Essentials
• VMware Telco Cloud Platform Essentials for RAN
• VMware Telco Cloud Platform RAN

Issue/Introduction

When viewing containers in a TKGm node, containerd incorrectly reports the status of a container as Running when its underlying process is not present on the node.

This can lead to nodes remaining stuck in Deleting state or cordoned state (SchedulingDisabled) because Kubernetes does not recognize that the container has been stopped.

 

From the Workload Cluster context, one or more of the following symptoms are observed:

• The corresponding pod for the affected container(s) is still showing as Running or Terminating:
  

  kubectl get pods -n <pod namespace> -o wide

   

• If the node is stuck Deleting, it will show as SchedulingDisabled state:
  

  kubectl get nodes

 

While SSH directly to the node where the pod and its container(s) is running:

• The incorrect container status will be reported in the output of crictl ps directly on the node:
  

  crictl ps

   

• Containerd logs report errors similar to the below:
  

  journalctl -xeu containerd
  
  MON DD HH:MM:SS <node name> containerd[988]: time="MON DD HH:MM:SS.ssssssZ" level=info msg="StopPodSandbox for \"<container ID>\""
  MON DD HH:MM:SS <node name> containerd[988]: time="MON DD HH:MM:SS.ssssssZ" level=error msg="StopPodSandbox for \"<container ID>\" failed" error="rpc error: code = DeadlineExceeded desc = failed to stop container \"<container ID>\": an error occurs during waiting for container \"<container ID>\" to be killed: wait container \"<container ID>\": context deadline exceeded"

Environment

TCA 3.4
TCP: 5.1

Cause

This is caused by a race condition between containerd's Exit and Exec probes.

Resolution

Containerd shows a container in Running state, however, the process does not actually exist on the node.

This issue is present in the following Tanzu kubernetes grid (TKGm) which use containerd v1.6.31 and it's resolved in the containerd v1.6.36 i v1.7.22.

For a Kubernetes 1.30 environment, the fix is included in the following image-builder-resource-bundle version:

• v1.30.13+vmware.1-tkg.1

Fixed Bundle Versions by Kubernetes Release

If you are running other Kubernetes versions under TKG v2.5.4, the issue is fixed in these respective bundles:

• K8s 1.33: v1.33.1+vmware.1-tkg.1
• K8s 1.32: v1.32.5+vmware.1-tkg.1
• K8s 1.31: v1.31.9+vmware.1-tkg.1
• K8s 1.30: v1.30.13+vmware.1-tkg.1
• K8s 1.29: v1.29.15+vmware.1-tkg.1
• K8s 1.28: v1.28.15+vmware.7-tkg.1
• K8s 1.27: v1.27.16+vmware.1-tkg.1

You can verify these component matrix mappings in the official release documentation:

• VMware Tanzu Kubernetes Grid v2.5.x Release Notes

Workaround
Restart contained directly on the affected node which will correctly update the status of all containers on the node:

systemctl restart containerd
systemctl restart kubelet