High Disk Usage in harbor-registry pod caused by _uploads directory

Products

• Tanzu Kubernetes Runtime

Issue/Introduction

The disk usage of the harbor-registry pod has increased significantly due to accumulated contents in the _uploads directory.

kubectl -n tanzu-system-registry exec -it harbor-registry-xxxx -c registry -- du -ah /storage | sort -hr | head -10
58G /storage/docker/registry/v2/repositories/###/###/_uploads      ###<<< showing high disk space usage

Environment

Harbor v2.9.1 from Tanzu Standard Package

Cause

uploadpurging feature periodically removes files from the _upload directory.

It is enabled by default.   Check the current value.

kubectl -n tanzu-system-registry exec harbor-registry-xxxx -c registry -- cat /etc/registry/config.yml | yq .storage.maintenance
 uploadpurging:
   enabled: true
   age: 168h     # Remove files in _upload directories which exist for a period of time, default is one week.
   interval: 24h # The interval of the purge operations	
   dryrun: false

In some environments, the default settings may not be sufficient to clean up the _uploads folder efficiently.

Currently, Tanzu Standard Package Harbor's values.yaml doesn't support tuning of the uploadpurging parameters.

The Resolution section will explain how to update the uploadpurging parameters by using the YTT overlay.

Resolution

1. Make the Harbor Registry Read Only mode

Harbor WebUI --> Administration --> Configuration --> System Settings --> Check "Repository Read Only" --> SAVE

Reference - Harbor Official Document

 

2. Generate a YTT Overlay YAML file

cat > pkg-harbor-overlay.yaml <<EOF
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")
#@ load("@ytt:yaml", "yaml")
 
#@ def add_purge_config(old, _):
#@   return yaml.encode(overlay.apply(yaml.decode(old), purge_config()))
#@ end
 
#@ def purge_config():
storage:
  maintenance:
    #@overlay/match-child-defaults missing_ok=True
    uploadpurging:
      enabled: true
      age: 1h
      interval: 1h
      dryrun: false
#@ end
 
#@overlay/match by=overlay.and_op(overlay.subset({"kind": "ConfigMap"}), overlay.subset({"metadata": {"name": "harbor-registry"}}))
---
data:
  #@overlay/replace via=add_purge_config
  config.yml:
EOF

 

3. Update the Harbor package

kubectl get pkgi -A | grep -E 'NAME|harbor'
#> NAMESPACE    NAME      PACKAGE NAME               PACKAGE VERSION         DESCRIPTION           AGE
#> mypackage    harbor    harbor.tanzu.vmware.com    2.9.1+vmware.1-tkg.1    Reconcile succeeded   6d22h
pkg_namespace=mypackage

# Update the package
tanzu -n ${pkg_namespace} package installed update harbor --ytt-overlay-file pkg-harbor-overlay.yaml
#> ...
#> 3:38:58PM: Deploy succeeded

 

4. Verify the result

# Check - all pods are "Running"
kubectl -n tanzu-system-registry get all

# Check - New uploadpurging settings
pod=$(kubectl -n tanzu-system-registry get pod -l app=harbor,component=registry -o jsonpath='{.items[0].metadata.name}')
kubectl -n tanzu-system-registry exec $pod -c registry -- cat /etc/registry/config.yml | yq .storage.maintenance
#> uploadpurging:
#>   enabled: true
#>   age: 1h
#>   interval: 1h
#>   dryrun: false

 

5. Revert the Harbor registry READONLY mode

Harbor WebUI --> Administration --> Configuration --> System Settings --> Uncheck "Repository Read Only" --> SAVE

Additional Information

Harbor parameter - registry.upload_purging