Supervisor pod is in ImagePullBackOff and kubelet reports "Failed to pull image" from localhost:5000 and "rpc error: code = NotFound desc = failed to pull and unpack image"

Products

• VMware vSphere Kubernetes Service
• VMware Tanzu Mission Control - SM
• VMware Tanzu Mission Control

Issue/Introduction

Scenario:

A pod expecting to be running on a Supervisor control plane node remains in ImagePullBackOff state.

For the pod in question, the kubelet reports that it "Failed to pull image" from localhost:5000 and "rpc error: code = NotFound desc = failed to pull and unpack image".

 

For example: 

The Supervisor control plane node reports that the tmc-agent-installer pod is in ImagePullBackOff state.

Looking closer at the pod, you see the following error:

Failed to pull image "localhost:5000/vmware/tmc-agent-installer:1.0": rpc error: code = NotFound desc = failed to pull and unpack image "localhost:5000/vmware/tmc-agent-installer:1.0": failed to resolve reference "localhost:5000/vmware/tmc-agent-installer:1.0": localhost:5000/vmware/tmc-agent-installer:1.0: not found

 

You confirm that you are able to manually pull the image from a working Supervisor control plane node.

crictl --debug pull localhost:5000/vmware/tmc-agent-installer:1.0

 

Output shows:

Image is up to date for sha256:#######################

 

However, you are unable to export that container image using ctr CLI in order to import to it to another node.

ctr -n k8s.io images export /tmp/tmc-agent-installer.tar "localhost:5000/vmware/tmc-agent-installer:1.0"

 

Output shows:

ctr: failed to get reader: content digest sha256:#######################: not found

Environment

VMware vSphere Kubernetes Service

Cause

The internal docker registry is served by nginx on localhost:5000.  However, this is an SSL port. 

Resolution

Utilize the HTTP proxied server port instead of the SSL port.  

 

Note: Here we will use the example tmc-agent-installer pod image referenced earlier.

 

• ssh to a Supervisor working control plane node VM

 

• Verify the value of the proxy_pass directive in the /etc/vmware/wcp/nginx/conf.d/XX-registry.conf file:

Example:

            proxy_pass                http://127.0.0.1:5002;

 

• Export the image over the proxy_pass port using the ctr CLI.  Port 5002 in this case.

ctr -n k8s.io images export /tmp/tmc-agent-installer.tar "localhost:5002/vmware/tmc-agent-installer:1.0"

 

• Transfer the exported image tar file to the destination Supervisor control plane node(s) that need it.

 

• ssh to the destination control plane node(s) 

 

• Import and retag the image to reference localhost:5000 instead of localhost:5002

ctr -n k8s.io image import --base-name localhost:5000/vmware/tmc-agent-installer:1.0 /tmp/tmc-agent-installer.tar

 

• Verify the image is correct on the destination control plane node(s).

crictl images | grep tmc-agent-installer

 

Example output:

localhost:5000/vmware/tmc-agent-installer                                               1.0                                   ##############       XX.XMB