update-node-password-runner pods in ImagePullBackOff state in Tanzu Kubernetes Release (TKr) 1.31.x and higher

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

• After updating a Tanzu Kubernetes Workload cluster to TKr 1.31.x, update-node-password-runner pods in the kube-system namespace fail to start and remain in ImagePullBackOff or CrashLoopBackOff status.

• kubectl describe pod on impacted pods shows the error:
  Back-off pulling image "localhost:5000/vmware.io/pause:3.9"
spec.initContainers{update-password}: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/" to rootfs at "/": create mountpoint for / mount: mountpoint "/run/containerd/io.containerd.runtime.v2.task/k8s.io
/update-password/rootfs" is on the top of rootfs "/run/containerd/io.containerd.runtime.v2.task/k8s.io/update-password/rootfs"

• Nodes in the guest cluster do not possess the 3.9 version of the pause image.

• Running crictl images | grep pause directly on a Guest Cluster Control Plane VM confirms only version 3.10 is available: localhost:5000/vmware.io/pause 3.10.

Environment

• vCenter Server 8.0 U3
• Tanzu Kubernetes Release (TKr) 1.31.x and later

Cause

The password update controller fails to reconcile the DaemonSet manifest after a cluster upgrade. This results in the update-node-password-runner referencing an incorrect pause image version (3.9 instead of 3.10) or utilizing an invalid host-path mount configuration.

Resolution

Follow the below steps to retrigger the password update controller and force the DaemonSet to align with the correct image version.

Method 1: Retrigger via Supervisor Annotation
Execute the below command from the Supervisor Cluster context to remove the timestamp annotation. This forces the controller to reconcile the cluster state and update the DaemonSet configuration.

1. Identify the cluster namespace and cluster name.

2. Run the annotation removal:

   kubectl annotate cluster -n <ClusterNamespace> <ClusterName> kubernetes.vmware.com/password-update-last-timestamp-
   

Method 2: Manual DaemonSet Modification
If the controller does not update the image version automatically after several minutes, manually edit the DaemonSet within the Guest Cluster context.

1. Edit the DaemonSet in the kube-system namespace:

   kubectl edit ds -n kube-system update-node-password-runner
   

2. Locate and update the image string from :3.9 to :3.10 in the following two locations within the manifest:

   • spec.template.spec.initContainers: Change image localhost:5000/vmware.io/pause:3.9 to localhost:5000/vmware.io/pause:3.10

   • spec.template.spec.containers: Change image localhost:5000/vmware.io/pause:3.9 to localhost:5000/vmware.io/pause:3.10

3. Save and exit. The pods will automatically restart using the correct image.