Harbor UI fails to open with error message "Core service is not available"

Products

• Tanzu Kubernetes Grid
• VMware Tanzu Kubernetes Grid 1.x
• VMware Tanzu Kubernetes Grid
• VMware vSphere with Tanzu
• Tanzu Mission Control

Issue/Introduction

The Harbor user interface is not accessible with error message like “Core service is not available”.

Besides this, the image pull requests start to fail.

Environment

This article applies to Harbor instance deployed on Kubernetes.

Cause

There could be numerous reasons which cause the Harbor core service down. We can review the pod log to understand the exact reason. One of the possibilities is that the pod, harbor-database-0, won't start.

The pod log can be retrieved through command like “kubectl logs -n harbor harbor-database-0”. We can see the disk resource is used up.

20xx-0x-xx 0x:xx:14.603 UTC [8] PANIC:  could not write to file “pg_wal/xlogtemp.8”: No space left on device 20xx-0x-xx 0x:xx:14.607 UTC [1] LOG:  startup process (PID 8) was terminated by signal 6: Aborted 20xx-0x-xx 0x:xx:14.607 UTC [1] LOG:  aborting startup due to startup process failure

 

Resolution

We can solve this issue by scaling up the persistent volume attached to the pod.

In Kubernetes v1.11 the persistent volume expansion feature is being promoted to beta. This feature allows users to easily resize an existing volume by editing the PersistentVolumeClaim (PVC) object. Users no longer have to manually interact with the storage backend or delete and recreate PV and PVC objects to increase the size of a volume. Shrinking persistent volumes is not supported.

PVC created from StorageClass can be edited to request more space. Kubernetes will interpret a change to the storage field as a request for more space, and will trigger automatic volume resizing. Here is an example,

More details about volume expansion can be found at the following links

1. Expanding Persistent Volumes Claims
2. Resizing Persistent Volumes using Kubernetes