Update-node-password-runner Pods Stuck In ImagePullBackOff on VKR 1.34.

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

The update-node-password-runner pods within a Tanzu Kubernetes Guest Cluster fail to start and remain in an ImagePullBackOff state.

Executing a kubectl describe pod command on the affected pods reveals an error indicating the inability to locate or pull the requested pause image version.

Events:
  Type     Reason   Age                From      Message
  ----     ------   ----               ----      -------
  Normal   Pulling  20m (x222 over 18h) kubelet  Pulling image "localhost:5000/vmware.io/pause:3.10"
  Warning  Failed   20m (x222 over 18h) kubelet  Failed to pull image "localhost:5000/vmware.io/pause:3.10": rpc error: code = NotFound desc = failed to pull and unpack image "localhost:5000/vmware.io/pause:3.10": failed to resolve reference "localhost:5000/vmware.io/pause:3.10": localhost:5000/vmware.io/pause:3.10: not found
  Normal   BackOff  24s (x4980 over 18h) kubelet  Back-off pulling image "localhost:5000/vmware.io/pause:3.10"
  Warning  Failed   24s (x4980 over 18h) kubelet  Error: ImagePullBackOff

Inspection of the node's available container images confirms only version 3.10.1 is present. This absence of version 3.10 directly correlates with the ImagePullBackOff events recorded in the pod description.

# crictl images | grep pause
localhost:5000/vmware.io/pause    3.10.1    4910dd319cfab    372kB

Environment

Tanzu Kubernetes Release (VKR) 1.34

Cause

There is a version mismatch regarding the required pause container image.

The pod specification explicitly requests a specific pause image version (e.g., 3.10), but the underlying Kubernetes node is provisioned with a different version (e.g., 3.10.1).

Resolution

Method 1: Retrigger via Supervisor Annotation

1. Identify the cluster namespace and cluster name.

2. Execute the following command from the Supervisor Cluster context to remove the timestamp annotation. This forces the controller to reconcile the cluster state and update the DaemonSet configuration:

   • kubectl annotate cluster -n <ClusterNamespace> <ClusterName> kubernetes.vmware.com/password-update-last-timestamp-

Method 2: Manual DaemonSet Modification

If the controller does not update the image version automatically, manually edit the DaemonSet within the Guest Cluster context.

1. Edit the DaemonSet in the kube-system namespace:

   • kubectl edit ds -n kube-system update-node-password-runner

2. Locate and update the image string to match the version provisioned on the underlying Kubernetes node in the following two locations within the manifest:

   • spec.template.spec.initContainers

   • spec.template.spec.containers

3. Save and exit. The pods will automatically restart using the correct image.