Guest Cluster update-node-password-runner pods in ImagePullBackOff status

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

Connecting to the guest cluster and running command kubectl get pods -n kube-system | grep -v Run, the update-node-password-runner pods have a status of ImagePullBackOff.

The guest cluster was recently upgraded.

Describing the impacted pods shows the following event:

Normal  BackOff  kubelet  Back-off pulling image "localhost:5000/vmware.io/pause:3.9"

Running crictl images | grep pause on the Guest Cluster Control Plane VM shows the following:

localhost:5000/vmware.io/pause          3.10   

Environment

vCenter 8.0 U3

Guest Cluster version 1.31

Cause

The password update task is attempting to use an outdated pause image for the KR being used. The KR is v1.31.11+vmware.3-fips for which the pause image version is 3.10 and not 3.9. 

Resolution

Open a case with Broadcom support.