Harbor disk expansion fails with an error "umount: /var/vcap/store: target is busy"

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

Symptoms:
When trying to increase the Harbor disk size by changing the value from Harbor Tile > Resource Config > Harbor-app's PERSISTENT DISK TYPE, it will fail while running "Apply Change".

The following is the error message outputted from the change log:

===== 2019-03-01 10:36:49 UTC Running "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=172.###.###.2 --deployment=harbor-container-registry-414b40a0ac638fb2c253 deploy /var/tempest/workspaces/default/deployments/harbor-container-registry-414b40a0ac638fb2c253.yml"
Using environment '172.###.###.2' as client 'ops_manager'

Using deployment 'harbor-container-registry-414b40a0ac638fb2c253'

 instance_groups:
 - name: harbor-app
-   persistent_disk_type: '20480'
+   persistent_disk_type: '51200'
Task 145

Task 145 | 10:36:51 | Preparing deployment: Preparing deployment (00:00:03)
Task 145 | 10:36:58 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 145 | 10:36:58 | Updating instance harbor-app: harbor-app/89abfc56-34e5-4680-8fbb-42fe9f9af616 (0) (canary) (00:12:51)
                   L Error: Action Failed get_task: Task e5c10582-bf20-4a70-594e-5762e1f8a37e result: Migrating persistent disk: Remounting persistent disk as readonly: Unmounting /var/vcap/store: Running command: 'umount /var/vcap/store', stdout: '', stderr: 'umount: /var/vcap/store: target is busy
       (In some cases useful info about processes that
        use the device is found by lsof(8) or fuser(1).)
': exit status 32
Task 145 | 10:49:49 | Error: Action Failed get_task: Task e5c10582-bf20-4a70-594e-5762e1f8a37e result: Migrating persistent disk: Remounting persistent disk as readonly: Unmounting /var/vcap/store: Running command: 'umount /var/vcap/store', stdout: '', stderr: 'umount: /var/vcap/store: target is busy
       (In some cases useful info about processes that
        use the device is found by lsof(8) or fuser(1).)
': exit status 32

Task 145 Started  Fri Mar  1 10:36:51 UTC 2019
Task 145 Finished Fri Mar  1 10:49:49 UTC 2019
Task 145 Duration 00:12:58
Task 145 error


Updating deployment:
 Expected task '145' to succeed but state is 'error'
Exit code 1
===== 2019-03-01 10:49:50 UTC Finished "/usr/local/bin/bosh --no-color --non-interactive --tty --environment=172.###.###.2 --deployment=harbor-container-registry-414b40a0ac638fb2c253 deploy /var/tempest/workspaces/default/deployments/harbor-container-registry-414b40a0ac638fb2c253.yml"; Duration: 780s; Exit Status: 1
Exited with 1.
Exited with 1.

Environment

Cause

If you log into the Harbor VM, you will find that /var/vcap/store/docker is still mounted.

harbor-app/89abfc56-34e5-4680-8fbb-42fe9f9af616:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4069472k,nr_inodes=1017368,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
:
:
/dev/sdb2 on /var/tmp type ext4 (rw,nosuid,nodev,noexec,relatime,data=ordered)
/dev/sda1 on /home type ext4 (rw,nodev,relatime,data=ordered)
/dev/sdc1 on /var/vcap/store type ext4 (rw,relatime,data=ordered)
/dev/sdc1 on /var/vcap/store/docker type ext4 (rw,relatime,data=ordered) <<<===!!!

This is why the associated script couldn't properly unmount /var/vcap/store.

The root cause of this issue is a code issue, where the monit script for Docker inside the Harbor VM has one missing step necessary for unmounting /var/vcar/store/docker.

Resolution

To workaround this issue, unmount /var/vcap/store/docker manually from the Harbor VM and rerun "Apply Change".

In other words, execute the following steps:

1. # umount /var/vcap/store/docker

2. Rerun "Apply Change"

The permanent fix will be implemented from Harbor 1.7.4 onward.