Containers are in Init:ErrImageNeverPull error state

Products

• VMware Aria Suite

Issue/Introduction

Symptoms:

• User is unable to do anything on the environment due to service not being up and running. Pods with Init:ErrImageNeverPull error on one or more nodes can be seen. Execute in order to see the states of the pods

kubectl get pods -n prelude

Example of pods with such error:

assessment-service-app-648bb587b4-24nw8        0/1     Init:ErrImageNeverPull   0          5h30m   10.244.0.235   prelude-004.eng.vmware.com   <none>           <none>
symphony-logging-daemonset-7phb9               0/1     ErrImageNeverPull        0          5h12m   10.244.0.239   prelude-004.eng.vmware.com   <none>           <none>
tango-blueprint-service-app-7c69f68d4c-2l8xg   0/1     Init:ErrImageNeverPull   0          5h28m   10.244.0.236   prelude-004.eng.vmware.com   <none>           <none>
tango-vro-gateway-app-7984985b88-rs7hs         0/1     Init:ErrImageNeverPull   0          5h35m   10.244.0.234   prelude-004.eng.vmware.com   <none>           <none>

Environment

VMware vRealize Automation 8.x

Cause

There might be different causes for this issue:

• Ephemeral storage in Prelude is 100% of the disk
• One or more of the storage Prelude disks are completely full or 80%+ full

(Disk /data is only 17% free, which is >80% used, which is a problem)
 

• Node has been restarted due to unhealthy node status

Resolution

Steps to recover from this state:

• Resize the affected disk and add to it at least 20GB, the more GBs added, the better
  • resizing happens through vSphere
• Reboot the affected node and wait some time for things to go into normal state again (about 30-50 mins)
  • Aria Automation documentation for increasing disk size indicates to also run the vracli disk-mgr resize command.

Alternative to the reboot is to execute the “/opt/scripts/restore_docker_images.sh” script on the affected node(s).