pinniped-concierge-kube-cert-agent is in ImagePullBackOff when upgrading Guest cluster to v1.31.4+vmware.1-fips

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• When upgrading Guest cluster to version v1.31.4, new CP nodes get deployed and joins the Guest cluster nodes list.
  
  
• All pods migrate to new node except pinniped-concierge-kube-cert-agent pod.
  
  
• On the new CP node, the pinniped-concierge-kube-cert-agent pod is in ImagePullBackOff with error in describe of pod as below:

Warning  Failed   11m (x4 over 12m)     kubelet  Failed to pull image "localhost:5000/tkg/packages/core/pinniped@sha256:<#################################>": rpc error: code = NotFound desc = failed to pull and unpack image "localhost:5000/tkg/packages/core/pinniped@sha256:<#################################>": failed to resolve reference "localhost:5000/tkg/packages/core/pinniped@sha256:<#################################>": localhost:5000/tkg/packages/core/pinniped@sha256:<#################################>: not found
  Warning  Failed   11m (x4 over 12m)     kubelet  Error: ErrImagePull
  Warning  Failed   11m (x6 over 12m)     kubelet  Error: ImagePullBackOff

• Image exists in the Guest cluster CP node gets listed when running below command:
  
  ctr -n k8s.io images list | grep pin

From above command we see sha<ID> for the image present on the CP node and sha<ID> of the image being pulled by pod does not match.

Environment

vSphere with Tanzu
VMware vSphere Kubernetes service

Cause

During the upgrade, the newly added control plane (CP) node have the pod deployment referencing image locations from the previous node. However, the image available on the guest cluster's CP node is of  newer version, resulting in the pod encountering an ImagePullBackOff error. 
The pod deployment points to old image reference resulting in replica sets getting deployed referencing the old image from deployment.

Resolution

• Review the pod's deployment and verify the image it is referencing under the 'image' field.
  
  kubectl get deployment -A | grep pinniped
kubectl get deployment pinniped-concierge-kube-cert-agent -n pinniped-concierge -o yaml | less


• List the image present in the new CP node:
  
ctr -n k8s.io images list | grep pinniped


• Compare both the above outputs, if the image referenced in the deployment is different from image present in the new CP node, contact Broadcom Support.