Snapshot-controller pod fails with ImagePullBackOff in TKGI cluster.

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Snapshot-controller pod in kube-system namespace fails with ImagePullBackOff in TKGI clusters in air-gapped environments.

snapshot-controller-1k5c8cb7c2-4eq3n              0/1     ImagePullBackOff   0             30m
snapshot-controller-25d67cb913-gzlkx              0/1     ImagePullBackOff   0             30m

Environment

Tanzu Kubernetes Grid Integrated Edition (TKGI ) 1.22.3

Cause

The Snapshot controller pod failed to run as the image [ gcr.io/k8s-staging-sig-storage/snapshot-controller:v7.0.2 ] is not found inside the worker node.

 

Resolution

However there is a workaround for this issue so we need to tag the registry.k8s.io to gcr.io image inside the worker node.

• ssh to the worker node and run the below command to tag the image.

ctr -n k8s.io images tag \
  registry.k8s.io/sig-storage/snapshot-controller:v7.0.2 \
  gcr.io/k8s-staging-sig-storage/snapshot-controller:v7.0.2

• Once the image is tagged the snapshot-controller pod will be in running state again.

Additional Information

Issue is fixed in TKGI 1.22.4.

Refer to resolved issues in TKGI 1.22.4 release notes. 

-> After upgrade to 1.22.4, in newly created cluster pod references registry.k8s.io image.. For upgraded clusters pod still references gcr.io image but we ensure that it loads correctly, preventing any image-related startup errors.
-> For clusters which are created in TKGi 1.22.3, after upgrade manually update the image URL.