Error: "ImagePullBackOff" for neutron-rabbit-init pod

Products

• VMware Integrated OpenStack

Issue/Introduction

• The neutron-rabbit-init pod fails to initialize and remains in an ImagePullBackOff state. Kubernetes pod events display the error Back-off pulling image "docker-registry.default.svc.viocm.local:5000/docker.io/rabbitmq:3.7-management".
• Manual docker pull attempts on the controller nodes fail.

Environment

7.x

Cause

The local VIO docker registry (vio-docker-registry) is unresponsive and unable to serve image manifests, or there is a local DNS/IP resolution mismatch preventing the worker nodes (controllers) from resolving the registry service address (docker-registry.default.svc.viocm.local).

Resolution

 

1. Log in to the vio-manager node via SSH.

2. Test the local registry responsiveness by attempting to pull the images manually:
   docker pull docker-registry.default.svc.viocm.local:5000/docker.io/rabbitmq:3.7-management
docker pull docker-registry.default.svc.viocm.local:5000/docker.io/rabbitmq:3.7.24

3. If the commands succeed on the manager, execute the first docker pull command directly on the affected controller nodes to cache the image locally.
   
   
4. If the commands fail on the manager, restart the registry deployment to clear hung processes:
   kubectl rollout restart deploy vio-docker-registry

5. Wait 2-3 minutes for the registry pods to initialize, then retry the docker pull commands from step 2 to confirm functionality.
   
   
6. (Optional) If the command output explicitly reports a failure to resolve docker-registry.default.svc.viocm.local to an IP address, retrieve the current Kubernetes service IP:
   kubectl get svc | grep registry

7. (Optional) Compare the returned service IP against the static entry for docker-registry.default.svc.viocm.local in /etc/hosts on the affected manager and controller nodes. Update /etc/hosts to correct any IP mismatches.