Snapshot-controller pod fails with ImagePullBackOff in TKGI cluster.

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Snapshot-controller pod in kube-system namespace fails with ImagePullBackOff in TKGI clusters in air-gapped environments.

snapshot-controller-1k5c8cb7c2-4eq3n              0/1     ImagePullBackOff   0             30m
snapshot-controller-25d67cb913-gzlkx              0/1     ImagePullBackOff   0             30m

Environment

Tanzu Kubernetes Grid Integrated Edition (TKGI ) 1.22.3

Cause

The Snapshot controller pod failed to run as the image [ gcr.io/k8s-staging-sig-storage/snapshot-controller:v7.0.2 ] is not found inside the worker node.

 

Resolution

This issue is fixed in TKGI 1.22.4 for any new cluster created under this version, and in 1.23.x for new and upgraded clusters.

Clusters upgraded to v1.22.4 can still experience this issue, however there is a workaround for this issue so we need to tag the registry.k8s.io to gcr.io image inside the worker node.

• ssh to the worker node and run the below command to tag the image.

ctr -n k8s.io images tag \
  registry.k8s.io/sig-storage/snapshot-controller:v7.0.2 \
  gcr.io/k8s-staging-sig-storage/snapshot-controller:v7.0.2

• Once the image is tagged the snapshot-controller pod will be in running state again. If the pod is still in the same state, ImagePullBackOff, then run: 

kubectl set image deployment/snapshot-controller snapshot-controller=registry.k8s.io/sig-storage/snapshot-controller:v7.0.2 -n kube-system

 

Additional Information

Refer to resolved issues in TKGI 1.22.4 release notes.