Tanzu Kubernetes Release 1.31 update-node-password-runner pods stuck in ImagePullBackOff due to pause image version mismatch

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

After updating a Tanzu Kubernetes Workload cluster to TKr 1.31.x, update-node-password-runner pods in the kube-system namespace fail to start and remain in ImagePullBackOff status.

Symptoms:

• kubectl describe pod on impacted pods shows the error: Back-off pulling image "localhost:5000/vmware.io/pause:3.9".

• Nodes in the guest cluster do not possess the 3.9 version of the pause image.

• Running crictl images | grep pause directly on a Guest Cluster Control Plane VM confirms only version 3.10 is available: localhost:5000/vmware.io/pause 3.10.

Environment

vCenter 8.0 U3
Tanzu Kubernetes Release (TKr) 1.31.x

Cause

The update-node-password-runner DaemonSet specification is hardcoded to reference an outdated pause image (version 3.9), which is no longer packaged or available in the TKr 1.31 image registry.

Resolution

Follow these steps to retrigger the password update controller and force the DaemonSet to align with the correct image version.

Method 1: Retrigger via Supervisor Annotation
Execute the following command from the Supervisor Cluster context to remove the timestamp annotation. This forces the controller to reconcile the cluster state and update the DaemonSet configuration.

1. Identify the cluster namespace and cluster name.

2. Run the annotation removal:

   kubectl annotate cluster -n <ClusterNamespace> <ClusterName> kubernetes.vmware.com/password-update-last-timestamp-
   

Method 2: Manual DaemonSet Modification
If the controller does not update the image version automatically after several minutes, manually edit the DaemonSet within the Guest Cluster context.

1. Edit the DaemonSet in the kube-system namespace:

   kubectl edit ds -n kube-system update-node-password-runner
   

2. Locate and update the image string from :3.9 to :3.10 in the following two locations within the manifest:

   • spec.template.spec.initContainers: Change image localhost:5000/vmware.io/pause:3.9 to localhost:5000/vmware.io/pause:3.10

   • spec.template.spec.containers: Change image localhost:5000/vmware.io/pause:3.9 to localhost:5000/vmware.io/pause:3.10

3. Save and exit. The pods will automatically restart using the correct image.