Harbor pods in imagepullbackoff due to change in Bitnami Catalogue

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• External harbor deployed in the guest cluster where the guest cluster is healthy but the pods are in imagepullbackoff.
  
  NAMESPACE                      NAME                                                       READY   STATUS                  RESTARTS       AGE
harbor                         harbor-core-###########-#####                              0/1     Init:ImagePullBackOff   0              17h
harbor                         harbor-jobservice-###########-#####                        0/1     Init:ImagePullBackOff   0              17h
harbor                         harbor-nginx-###########-#####                             0/1     Init:ImagePullBackOff   0              17h
harbor                         harbor-portal-###########-#####                            0/1     Init:ImagePullBackOff   0              17h
harbor                         harbor-postgresql-#                                        0/1     ImagePullBackOff        0              17h
harbor                         harbor-redis-master-#                                      0/1     ImagePullBackOff        0              17h
harbor                         harbor-registry-###########-#####                          0/2     Init:ImagePullBackOff   0              17h
harbor                         harbor-trivy-#                                             0/1     Init:ImagePullBackOff   0              17h
  
  
• Describe of harbor pods shows below error message:
  Message: Backoff pulling image "docker.io/bitnami/harbor-core:#.##.#-debian-##-r##"
 
• The describe of pod deployment shows image points to "docker.io/bitnami/harbor..."

Environment

vSphere Kubernetes Service 8.X

Cause

Beginning August 28, all current Bitnami container images — including older versions like 2.50.0 and 10.6 — has been moved from docker.io/bitnami to docker.io/bitnamilegacy over a two-week period.

Resolution

Steps to point the image location in pod deployments to "docker.io/bitnamilegacy/harbor ..":

1. Take a backup of deployment:
   kubectl get deployment -n harbor <deployment name> -o yaml > <deployment-name>-deployment.yaml
   
   
2. Scale the deployment to 0. [Verify the deployment number before scaling to 0]:
   kubectl scale deployment harbor-<name> --replicas=0 -n harbor
   
   
3. Edit the deployment image location to "docker.io/bitnamilegacy/harbor .." instead of "docker.io/bitnami/harbor .." and save:
   kubectl edit deployment -n harbor <deployment name> -o yaml
   
   
4. Verify the changes were made and saved:
   kubectl get deployment -n harbor <deployment name> -o yaml | less
 
5. Scale back up the deployment:
   kubectl scale deployment harbor-<name> --replicas=# -n harbor
   
   
6. List the replicasets and verify the 1/1 status for newly deployed replicaset:
   kubectl get rs -n harbor
   
   
7. Delete the old replicasets which still point to old image location with "bitnami":
   kubectl delete rs <old-rs-name> -n harbor
   
   
8. Check the pods and verify none of the pods are in ImagePullBackOff.
   
   
9. For some pods which do not have any deployments or replicasets corresponding to the pod - harbor-postgresql-#, harbor-redis-master-#, harbor-registry-###########-#####, harbor-trivy-#, we require to edit the pods and point to correct image location (with "docker.io/bitnamilegacy/harbor..").
   
   
10. Confirm all the harbor pods are up and running.
    kubectl get pods -n harbor
NAME                                 READY   STATUS    RESTARTS      AGE
harbor-core-###########-#####         1/1     Running   9 (15m ago)   32m
harbor-jobservice-###########-#####   1/1     Running   6 (11m ago)   19m
harbor-nginx-###########-#####        1/1     Running   0             61m
harbor-portal-###########-#####       1/1     Running   0             42m
harbor-postgresql-#                   1/1     Running   0             19h
harbor-redis-master-#                 1/1     Running   0             19h
harbor-registry-###########-#####     2/2     Running   0             40m
harbor-trivy-#                        1/1     Running   0             19h
    
    

Additional Information

• For more details refer:
  Broadcom Introduces Bitnami Secure Images For Production-Ready Containerized Applications - Broadcom News and Stories
  bitnami/harbor-core - Docker Image | Docker Hub