Error : "Back-off pulling image "vmware/calico/cni:<version>" observed when during calico pods failure.

Products

• VMware Integrated OpenStack
• VMware Telco Cloud Platform

Issue/Introduction

• Controller node unresponsive to standard health checks.
• Kubernetes pods, specifically Calico CNI and RabbitMQ, fail to initialize or report errors.
• Kubernetes events report FailedMount errors.
• Calico pods are stuck in an ImagePullBackOff error state.

Environment

VIO 7.3

TCP 2.5  

Cause

•  The container images are missing from the the local node's bootstrap cache and  Calico CNI pods cannot pull required images.
•  RabbitMQ relies on  inter-pod communication and DNS resolutionand this networking failure prevents RabbitMQ pods from forming a cluster and joining the primary messaging service.

Resolution

Manually copy the missing  bootstrap images from VIO Manager and reload the image.

1. Log in to the VIO manager node.
2. Locate the bootstrap archive at 
   
   /opt/vmware/data/bootstrap-images/bootstrap-images.tar.
3. Copy this file to the affected controller node(s).
   
   scp -i /opt/vmware/data/bootstrap-images/bootstrap-images.tar vioadmin@<CONTROLLER_IP>:/tmp/
4. On the affected controller node, run the following command to load the images: 
   
   docker load -i /tmp/bootstrap-images.tar
5. Allow a few minutes for the Kubernetes service to detect the now-available images and reconcile the pod states.
6. Verify that all services are running correctly by executing: 
   
   viocli get deployment

Once the images are loaded, the Calico pods should transition to a Running.