Harbor upgrade to 1.10.3 is failing
Article ID: 316800
Updated On:
Products
Issue/Introduction
Symptoms:
· You have applied the work around for sudo root password expiry issue as per the this KB Article
· When you try to upgrade the PKS to 1.10.3, it fails with harbor pre-start script:
Action Failed get_task: Task cfa31171-922a-4750-4174-f2a6345b252c result: 1 of 4 pre-start scripts failed. Failed Jobs: harbor. Successful Jobs: enable-bosh-dns, bosh-dns, wavefront."
· You may also see the error similar to:
Action Failed get_task: Task 3728ecc4-152d-46a0-7cdf-a46ff4ed5d70 result: Unmounting persistent disk: Running command: 'umount /dev/sdc1', stdout: '', stderr: 'umount: /var/vcap/store/docker: target is busy
· In the /var/vcap/sys/logs/harbor/pre-start.stderr.log file on the Harbor vm, you see the entries similar to:
ERROR: for harbor-log Cannot create container for service log: Conflict. The container name "/harbor-log" is already in use by container "33f858257267241bcb59b2aa69a360973079be9171d3697773a8fdce4c711b24". You have to remove (or rename) that container to be able to reuse that name.
Environment
Cause
This issue occurred due to the previous workaround applied for sudo root password expire issue. We had manually created the harbor-log container earlier, so docker job is busy and failed to unmount.
Resolution
To resolve this issue:
1. Log in to the Harbor VM from the bosh by running the command:
bosh ssh <harbor vm > -d <deployment name>
2. Switch to sudo mode by running the command and setup alias for docker:
sudo -i
alias docker='/var/vcap/packages/docker/bin/docker -H unix:///var/vcap/sys/run/docker/dockerd.sock'
3. Remove all docker containers:
docker stop $(docker ps -aq) && docker rm $(docker ps -aq)
4. Stop docker daemon:
/var/vcap/jobs/docker/bin/ctl stop
# check docker daemon is stopped
ps -ef|grep docker
5. Check all sub directory under /var/vcap/store is umounted
cat /proc/mounts |grep /var/vcap/store
# make sure /var/vcap/store/docker is not be mounted, if there is a sub directory, umount the sub directory first.
umount /var/vcap/store/docker/<path to sub directory>
umount /var/vcap/store/docker
6. Click apply-changes in Ops Manager to upgrade Harbor Tile to 1.10.3.
Additional Information
Feedback
