Recover Deleted Images from Node Cache

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

In the event images are deleted from Harbor by retention policies, there are ways to recover the images. If the images are still running on K8s nodes, it may be possible to get the image via the node local image cache.

Resolution

To get all images on all nodes in the cluster, you can run the following on each cluster:

bosh -d <cluster_dep> ssh -c "crictl images --digest" to get all 

https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/

You'll have to use something like ctr (which ships in the containerd release) or nerdctl, by uploading the CLI binary to the workers and running commands over SSH to push the images back to Harbor.

Note: Pause any K8s upgrades, and app teams shouldn't perform any kind of deployments, such as changing environment variables.