Unable to increase the Harbor disk size from Ops Manager

Products

• VMware

Issue/Introduction

Symptoms:

• Increasing the Harbor disk size from Ops Manager is failing.

• In the Ops Manager change log, you see the entries similar to:

Task 1181 | 13:13:22 | Preparing deployment: Preparing deployment (00:00:02)
Task 1181 | 13:13:26 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 1181 | 13:13:26 | Updating instance harbor-app: harbor-app/ab986a44-3f83-4679-bad5-3b7d127e5c87 (0) (canary) (00:12:14)
L Error: Action Failed get_task: Task 29a18b11-9888-4378-7e59-fdbb3f592716 result: Migrating persistent disk: Remounting persistent disk as readonly: Unmounting /var/vcap/store: Running command: 'umount /var/vcap/store', stdout: '', stderr: 'umount: /var/vcap/store: target is busy
(In some cases useful info about processes that
use the device is found by lsof(8) or fuser(1).)
': exit status 32
Task 1181 | 13:25:40 | Error: Action Failed get_task: Task 29a18b11-9888-4378-7e59-fdbb3f592716 result: Migrating persistent disk: Remounting persistent disk as readonly: Unmounting /var/vcap/store: Running command: 'umount /var/vcap/store', stdout: '', stderr: 'umount: /var/vcap/store: target is busy
(In some cases useful info about processes that
use the device is found by lsof(8) or fuser(1).)
': exit status 32

Task 1181 Started Mon Jan 28 13:13:22 UTC 2019
Task 1181 Finished Mon Jan 28 13:25:40 UTC 2019
Task 1181 Duration 00:12:18
Task 1181 error
Updating deployment:
Expected task '1181' to succeed but state is 'error'
Exit code 1

Environment

VMware PKS 1.x

Cause

This issue occurs due to an error while un-mounting the existing which is still in use by some of the Harbor processes.

Resolution

This is a known issue with Harbor 1.5.x,1.6.x, and 1.7.x will be resolved in future versions.

Workaround:
Notes:

This workaround can be done before click "Apply-Changes" to save time.

Before "Apply-Changes", need to exit the console login to harbor VM.

To work around this issue:
 

1. Log in to the Harbor VM from the bosh by running the command:
   bosh ssh <harbor vm > -d <deployment name>
    
2. Switch to sudo mode by running the command: sudo -i
3. Check the current mount on /var/vcap/store and its sub folder:
   # cat /proc/mounts |grep /var/vcap/store
   
   /dev/sdc1 /var/vcap/store ext4 rw,relatime,data=ordered 0 0
   /dev/sdc1 /var/vcap/store/docker ext4 rw,relatime,data=ordered 0 0
   
   Note: Above output confirms docker is still mounted to existing disk.
    
4. To unmount this, stop the harbor and dockerd services.
   monit stop harbor
   monit stop dockerd
   Note: If monit commands failed, then run docker stop $(docker ps -aq) && docker rm $(docker ps -aq) to cleanup all the containers. 
5. Unmount all sub directories. Ensure to unmount all underlying directory first, do not umount /var/vcap/store.
   
   # umount /var/vcap/store/docker
    
6. Run the below command again to make confirm /var/vcap/store/docker is unmounted.
   # cat /proc/mounts \|grep /var/vcap/store
   
   /dev/sdc1 /var/vcap/store ext4 rw,relatime,data=ordered 0 0
    
7. Then click the Apply Changes from the Ops Manager to increase the harbor disk size.