Harbor apply change failed with error "'umount /dev/sdc1', stdout: '', stderr: 'umount: /var/vcap/store: target is busy".

Products

• VMware Tanzu Kubernetes Grid Integrated Edition

Issue/Introduction

If Habor is failing with ceritifcate error and tried to recreate the VM may leads to this error.

Error: Action Failed get_task: Task dfcf5670-1d0e-5432-5685-c6731d7dd5da result: Unmounting persistent disk: Running command: 'umount /dev/sdc1', stdout: '', stderr: 'umount: /var/vcap/store: target is busy.

Verify the mounts as mentioned in the below KB.

https://knowledge.broadcom.com/external/article/298550/harbor-disk-expansion-fails-with-an-erro.html

If you do not see the below mount then follow this KB

/dev/sdc1 on /var/vcap/store/docker type ext4 (rw,relatime,data=ordered) <<<===!!!

Environment

TKGI with Habor 

Cause

If you log into the Harbor VM, you will find that below process are still using the files or directories under /var/vcap/store

harbor-app/dfcf5670-1d0e-5432-5685-c6731d7dd5da:/var/vcap/sys/log/harbor# fuser -mv /var/vcap/store

                     USER        PID ACCESS COMMAND

/var/vcap/store:     root     kernel mount /var/vcap/store

                     root       5911 F...m dockerd

                     root       5934 F...m containerd

 

 

Resolution

Kill the process which are using the files or directories under /var/vcap/store

harbor-app/dfcf5670-1d0e-5432-5685-c6731d7dd5da:/var/vcap/sys/log/harbor# kill -9 5911

harbor-app/dfcf5670-1d0e-5432-5685-c6731d7dd5da:/var/vcap/sys/log/harbor# kill -9 5934

bash: kill: (5934) - No such process

harbor-app/dfcf5670-1d0e-5432-5685-c6731d7dd5da:/var/vcap/sys/log/harbor# fuser -mv /var/vcap/store

                     USER        PID ACCESS COMMAND

/var/vcap/store:     root     kernel mount /var/vcap/store