Harbor replication job fails at 75%

Products

• VMware vSphere Kubernetes Service

Issue/Introduction

• Harbor registry deployed from Standard Packages in VKS cluster. See Install the Harbor Registry Using the Tanzu CLI
• Replication job fails in this Harbor registry at 75% with different error each time by checking the Logs of each replication job.
• The pod harbor-jobservice logs indicate the /var/log/jobs/ is full:
  
  [YYYY-MM-DDTHH:MM:SS]Z [ERROR] [/jobservice/runner/redis.go:123]: Job 'REPLICATION:<job-id>' exit with error: open /var/log/<job-id>.log: no space left on device

Environment

vSphere Kubernetes Services

Resolution

To resolve the issue:

1. Login VKS cluster. See Connect to a TKG Service Cluster as a vCenter Single Sign-On User with Kubectl
2. Get the pod name of harbor-jobservice:
   
   # kubectl get pod -n tanzu-system-registry


3. Go into the pod and check the disk usage is full:
   
   # kubectl exec -it -n tanzu-system-registry harbor-jobservice-<id>
$ df -h
...
/dev/sdf        973M  906M     0 100% /var/log/jobs
...


4. Delete those large size and old log files to release space.
5. Create a new replication job which should be able to run completely.