Pulling images using proxy from external registries fail
Article ID: 345064
Updated On:
Products
Issue/Introduction
To help analyse and troubleshoot issues with image pull and proxy.
Symptoms:
While trying to run Conformance or Lite test from the TMC.
Pods can't start to create because an error: ErrImagePull. The traffic to the Internet is through the proxy which is configured.
Curl to this repository address from "clean" linux VM which is connected to the worker network is successful.
That means, the traffic through proxy is OK but something is wrong or not working as expected.
Errors of type can be seen when describe pod:
From Worker node:
[ /etc/containerd ]# crictl --debug pull registry.k8s.io/e2e-test-images/agnhost:2.43
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:registry.k8s.io/e2e-test-images/agnhost:2.43,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,}
E0703 13:08:55.109471 14896 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"registry.k8s.io/e2e-test-images/agnhost:2.43\": failed to resolve reference \"registry.k8s.io/e2e-test-images/agnhost:2.43\": failed to do request: Head \https://europe-north1-docker.pkg.dev/v2/k8s-artifacts-prod/images/e2e-test-images/agnhost/manifests/2.43\: authenticationrequired" image="registry.k8s.io/e2e-test-images/agnhost:2.43"
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "registry.k8s.io/e2e-test-images/agnhost:2.43": failed to resolve reference "registry.k8s.io/e2e-test-images/agnhost:2.43": failed to do request: Head https://europe-north1-docker.pkg.dev/v2/k8s-artifacts-prod/images/e2e-test-images/agnhost/manifests/2.43: authenticationrequired
Cause
As per links below:
https://github.com/kubernetes/registry.k8s.io/blob/main/docs/debugging.md
The registry.k8s.io is a Kubernetes container images registry that behaves generally like an OCI compliant registry. Since registry.k8s.io is a proxy routing traffic to the closest available source, you will need connectivity to several domains to download images. It is also best for performance to create your own registry mirror.
Verify image pulls
Since registry.k8s.io proxies image components to the nearest source, you should validate the ability to pull images. The ability to pull images should be tested on the machine running the image which will often be a node in your Kubernetes cluster. The location where you pull image components from depends on the source IP address of the node.
You may use commands such as crane, oras, crictl or docker to verify the ability to pull an image. If you run the command crane pull --verbose registry.k8s.io/pause:3.9 pause.tgz for example, you will see it query registry.k8s.io first and then at least two other domains to download the image. If things are working correctly and you ran crane pull --verbose registry.k8s.io/pause:3.9 pause.tgz 2>&1 | grep 'GET https' (from Colorado):
2023/03/17 04:45:48 --> GET https://registry.k8s.io/v2/
2023/03/17 04:45:48 --> GET https://registry.k8s.io/v2/pause/manifests/3.9
2023/03/17 04:45:48 --> GET https://us-west1-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9
2023/03/17 04:45:48 --> GET https://registry.k8s.io/v2/pause/manifests/sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10
2023/03/17 04:45:48 --> GET https://us-west1-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10
2023/03/17 04:45:49 --> GET https://registry.k8s.io/v2/pause/blobs/sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c
2023/03/17 04:45:49 --> GET https://prod-registry-k8s-io-us-west-2.s3.dualstack.us-west-2.amazonaws.com/containers/images/sha256%3Ae6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c
2023/03/17 04:45:49 --> GET https://registry.k8s.io/v2/pause/blobs/sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007 [body redacted: omitting binary blobs from logs]
2023/03/17 04:45:49 --> GET https://prod-registry-k8s-io-us-west-2.s3.dualstack.us-west-2.amazonaws.com/containers/images/sha256%3A61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007 [body redacted: omitting binary blobs from logs]
From current location, the pull command accesses registry.k8s.io, us-west1-docker.pkg.dev and prod-registry-k8s-io-us-west-2.s3.dualstack.us-west-2.amazonaws.com. You will need to have DNS and HTTP access to these domains on your node to pull images.
Tests from different geolocations shows different sites for the images to be pulled
registry.k8s.io
us-west2-docker.pkg.dev
prod-registry-k8s-io-us-west-1.s3.dualstack.us-west-1.amazonaws.com
registry.k8s.io
europe-north1-docker.pkg.dev
prod-registry-k8s-io-eu-north-1.s3.dualstack.eu-north-1.amazonaws.com
To confirm the endpoints blocked or having problem a packet capture can be used or the output of the circtl pull can be verified as provided in the symptoms.
Resolution
Adding the geolocations or adding a wildcard can help mitigate the problem where the proxy is allowing only main site but not the redirected sites from where the actual images reside.
List of all registries that might take place for running conformance tests from TMC
| RegistryList | |
| GcAuthenticatedRegistry: "gcr.io/authenticated-image-pulling", | |
| PromoterE2eRegistry: "registry.k8s.io/e2e-test-images", | |
| BuildImageRegistry: "registry.k8s.io/build-image", | |
| InvalidRegistry: "invalid.registry.k8s.io/invalid", | |
| GcEtcdRegistry: "registry.k8s.io", | |
| GcRegistry: "registry.k8s.io", | |
| SigStorageRegistry: "registry.k8s.io/sig-storage", | |
| PrivateRegistry: "gcr.io/k8s-authenticated-test", | |
| MicrosoftRegistry: "mcr.microsoft.com", | |
| DockerLibraryRegistry: "docker.io/library", | |
| CloudProviderGcpRegistry: "registry.k8s.io/cloud-provider-gcp", |
Additional Information
https://docs.aws.amazon.com/general/latest/gr/s3.html
Impact/Risks:
Certain images cannot be pulled due to a block to mirrored image repositories where main site is redirecting the client.
Feedback
