Pushing image to Harbor registry results in error "received unexpected HTTP status: 500 Internal Server Error"

Products

• VMware Tanzu Kubernetes Grid

Issue/Introduction

Symptoms:
When trying to push image to Harbor registry We get the error "received unexpected HTTP status: 500 Internal Server Error"

For reference see below o/p :

ubuntu@jumpbox:~$ docker push harbor.pks-33.slot-59.pez.vmware.com/library/store/appdynamics/machine:4.4
The push refers to repository [harbor.pks-33.slot-59.pez.vmware.com/library/store/appdynamics/machine]
21b2fc203dbe: Retrying in 1 second
d07db587097b: Retrying in 1 second
2cb3ad46827d: Retrying in 1 second
2cfa48cdb08b: Retrying in 1 second
27860a0ae428: Retrying in 1 second
60e83c8e2fcc: Waiting
983855c885d4: Waiting
cdb3f9544e4c: Waiting
received unexpected HTTP status: 500 Internal Server Error

Environment

VMware Tanzu Kubernetes Grid 1.x

Cause

Check the logs of the pod harbor-registry-xxxxxx by running 

kubectl logs harbor-registry-xxxxxxx-xxxx -c registry -n tanzu-system-registry | less

For reference :

kubectl logs harbor-registry-769bfc647d-wbg7p -c registry -n tanzu-system-registry | less
 

For reference see below o/p :
We can see from the logs, it says "no space left on device"

time="2021-09-14T05:48:20.646202184Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user"
err.code=unknown err.detail="filesystem: mkdir /storage/docker/registry/v2/repositories/library/store/appdynamics/machine/_uploads/2f498863-b421-4ca1-a32f-46de4188996c: no space left on device" 

err.message="unknown error" go.version=go1.13.8 http.request.host=harbor.pks-33.slot-59.pez.vmware.com http.request.id=14fa25b4-55dc-4bf8-9cea-49888cca16ed http.request.method=POST http.request.remoteaddr=10.213.48.132  

 

To confirm the same run the below command to check the filesystem utilization for pod harbor-registry-xxxxxxx-xxxx point it to container registry.

Running kubectl exec harbor-registry-xxxxxxx-xxxx -n tanzu-system-registry -c registry -- df -h 

For reference see below o/p : 
kubectl exec harbor-registry-769bfc647d-wbg7p -n tanzu-system-registry -c registry -- df -h 

Filesystem     Size  Used Avail Use% Mounted on
overlay         39G   10G   27G  28% /
tmpfs           64M     0   64M   0% /dev
tmpfs          3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/sdc        10G  9.2G     0  100% /storage
/dev/sda3       39G   10G   27G  28% /etc/hostname
shm             64M     0   64M   0% /dev/shm
tmpfs          3.8G  4.0K  3.8G   1% /etc/registry/passwd
tmpfs          3.8G   12K  3.8G   1% /etc/harbor/ssl/registry
tmpfs          3.8G   12K  3.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs          3.8G     0  3.8G   0% /proc/acpi
tmpfs          3.8G     0  3.8G   0% /proc/scsi
tmpfs          3.8G     0  3.8G   0% /sys/firmware

We can clearly see that /storage is showing 100% used.

Resolution

The disk space exhaustion should be addressed using the workarounds highlighted below in order to solve this issue.

Workaround:
To resolve this issue,

Workaround 1: Remove the old images that are not in use and run 'Garbage Collection' from harbor GUI
Workaround 2:- To expand PV and PVC size: https://kb.vmware.com/s/article/82181

Please note: When you delete images from Harbor, space is not automatically freed up. You must run garbage collection to free up space by removing blobs that are no longer referenced by a manifest from the file system.

Refer to Official docs of Harbor: https://goharbor.io/docs/1.10/administration/garbage-collection/

Additional Information

Impact/Risks:
Due to out of space issue on the container's filesystem, no more images can be pushed.