Control Plane Failures Due to Missing Local Image pause:3.4.1

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

In a TKG Service cluster, core control plane components such as etcd, kube-apiserver, kube-scheduler, and antrea fail to start due to a missing container image. Affected pods logged the following error:

failed to do request: Head "http://localhost:5000/v2/vmware.io/pause/manifests/3.4.1": dial tcp 127.0.0.1:5000: connect: connection refused

This indicates that the container runtime attempted to resolve a locally tagged image (localhost:5000/vmware.io/pause:3.4.1) that was no longer available.

Environment

VMware vSphere with Tanzu

Cause

During an image cleanup operation, the locally tagged pause:3.4.1 image was inadvertently removed. This image was expected to be resolved from the container runtime’s local cache and was not configured to be pulled from an external registry. Its absence caused any dependent pods to fail to start.

Resolution

1. SSH into the affected control plane node(s).
2. Pull the required image:
   • sudo crictl pull registry.k8s.io/pause:3.4.1
3. Re-tag the image locally:
   • sudo crictl -n k8s.io image tag registry.k8s.io/pause:3.4.1 localhost:5000/vmware.io/pause:3.4.1
4. Confirm that affected pods restart and services recover.