Error: "Core service is not available"

Products

• VMware Telco Cloud Automation
• VMware Telco Cloud Platform

Issue/Introduction

• unable to access the CNF harbor, getting error message in GUI

• Running kubectl logs -n <harbor-namespace> <harbor-pod> can show:
  • PANIC: could not write to file
  • No space left on device

Environment

TCA: 3.3

TCP: 5.0

Cause

Space is filled up on the data disk, this can be confirmed by accessing harbor and running:

df -h

Example:

Filesystem    Size  Used Avail Use% Mounted on
/dev/sdb#      80G   80G   80G 100% /dev

Resolution

Check Directory Filesystem for large files:

1. Use the following (directory that is filled from df -h):
   1. To identify which directory has large files from SSH on harbor run:
      1. du -h ./ -d 1
      2. from above output you can then determine if logs are filled or if a different directory is filled.
      3. If /var/log/harbor/ is filled you can remove them safely using:
         
         1. sudo find /var/log/harbor/ -name "*.log" -exec truncate -s 0 {} +
         2. check that enough space is freed up.
         3. To Mitigate further issue with logs taking up disk space look at the following for log rotation.
         4. If you wish to proceed to increase disk space continue with next steps to resize disk.
      4. If /data/registry is filled these are system images and you can proceed to resize disk. or:
         1. If harbor is a standalone VM you can run:
            
            1. docker system prune -f
               NOTE: This will not delete your Harbor database or image registry, but it will clean up old build caches and stopped containers.
         2. If cleanup does not resolve your disk space issue, you will want to continue to resize disk.

Resize Disk on Photon OS:

1. SSH to Harbor
2. To identify hard disk size run:
   lsblk
3. In vCenter server UI find the virtual machine and disk that matches the size from prior command:
   1. Open VMware vSphere Client or ESXi host client.
   2. Right-click the virtual machine.
   3. Click Edit Settings.
   4. Select Hard Disk.
   5. Increase the size of the disk.

4. Note: If this option is greyed out, the disk may be running on snapshots or the disk may be at the maximum allowed size depending on the block size of the datastore.

5. Using Expanding Disk Partition procedure steps are as follows:
   
   1. On SSH Session to harbor run (NOTE: # needs to be replaced with block id found from lsblk reference Step 5 link for example):
      1. echo 1 > /sys/class/block/sd#/device/rescan
      2. tdnf install parted
      3. parted /dev/sd#
         You will see a prompt (parted) along with disk partition information 
      4. identify from above command which flagged disk number partition is needed and run:
         1. resizepart # 100%
         2. resize2fs /dev/sda2
      5. Rerun lsblk and df -h to confirm disk is resized.
         
         
6. Once disk size is addressed the UI will be accessible again without the error on login.

If running Harbor on a different OS please see the following:

Increasing the disk size on a Virtual Machine

Additional Information