TKGi - Can not login Harbor web UI due to "Core service is not available" - Persistent Disk Full

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

TKGi Harbor web UI login failed with "Core service is not available" error.

Environment

Tanzu Kubernetes Grid Integrated Edition

Cause

There could be numerous reasons for the harbor core service to be down. One possibility is that the disk full of Persistent Disk.

# Check disk usage --> Found disk full
DEPLOYMENT=$(bosh ds --column=Name | grep harbor-container-registry)
bosh -d ${DEPLOYMENT} ssh harbor-app/0 "df -h"
#> Filesystem      Size  Used Avail Use% Mounted on
#> ...
#> /dev/sdc1        2.0T  1.9T  0   100% /var/vcap/store

Resolution

Expand the Persistent Disk size via Ops Manager (Check actual behavior in "Additional Information" section).

 

1. Check the free space of the datastore

If the current persistent disk size is 2TB, an additional 2TB free space is required in the same datastore.
If there isn’t enough free space in the datastore, please expand the datastore size in advance.

 

2. Expand the Persistent Disk

• 2-1: Opsman web UI --> Harbor Tile --> Settings --> Resource Config
• 2-2: Select a bigger PERSISTENT DISK TYPE and SAVE
  • New virtual disk is created as "Thin" provisioning
• 2-3: APPLY CHANGE
  • Uncheck TKGI tile
  • Uncheck "smoke-testing" in the Harbor tile
• 2-4: Wait for completing the task
  • If the actual data size in the persistent disk is over 1TB, it takes 2-3 hours

 

3. (optional) Make disk space

After logging in to the harbor web UI, kick the "Garbage Collection" to make disk space.

Administration --> Clean Up --> "GC NOW"

Additional Information

# Study: What happens after kicking to expand the Persistent Disk?

1. Attach a new virtual disk for the harbor VM 
2. Copy the data from the existing disk (/dev/sdc1) to new disk (/dev/sdd1)
3. Delete the existing disk
4. Start harbor-app

 

# Best Practice

• Schedule Garbage Collection
• Configure Project Quotas (Don't use "unlimited")